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Abstract 

The  management  of  United  States  Air  Force  Automatic  Test  Systems  presents  a 
variety  of  issues  that  affect  the  long-term  capability  and  mission  readiness  of  weapon 
systems.  Historical  procurement  processes  suggest  that  individual  System  Program 
Offices  developed  and  replaced  Automatic  Test  Equipment  “as-required”;  their  specific 
managerial  actions  being  driven  in  parallel  support  of  the  managed  weapon  system.  With 
the  passage  of  time,  growing  numbers  of  platform-specific  pieces  of  Automatic  Test 
Equipment,  coupled  with  aging  aircraft,  led  to  significant  increases  in  parts  obsolescence 
and  greater  sustainment  challenges.  This  obsolescence  and  a  desire  for  longer-term 
viability  provided  the  inertia  for  a  movement  towards  consolidated,  common  test 
equipment  that  would  support  multiple  weapon  systems. 

In  1998,  the  Air  Force  Chief  of  Staff  directed  Headquarters  Air  Force  Materiel 
Command  to  establish  a  support  equipment  office  to  facilitate  the  combination  of  similar 
legacy  Automatic  Test  Equipment  into  common  equipment.  Still,  two  different 
sustainment  strategies  continue  to  exist  today.  Some  Automatic  Test  Equipment 
continues  to  be  primarily  supported  by  individual  System  Program  Offices,  while  other 
Automatic  Test  Equipment  is  primarily  managed  by  the  Automated  Test  Systems 
Division  at  Warner-Robins  Air  Logistics  Center  (WR-ALC/LEA).  This  research  is 
designed  to  examine  the  similarities  and  differences  of  these  two  approaches. 

Specifically,  this  research  focuses  on  the  management  of  two  pieces  of  Automatic  Test 
Equipment  for  a  mature  aircraft  system  -  the  F-15. 
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EVALUATING  MANAGEMENT  STRATEGIES 


FOR  AUTOMATED  TEST  SYSTEMS/EQUIPMENT  (ATS/E): 
ANF-15  CASE  STUDY 


I.  Introduction 


Overview 

The  management  of  United  States  Air  Force  Automatic  Test  Systems  and 
Equipment  presents  a  variety  of  supportability  and  sustainability  issues  that  directly 
affect  the  long-term  capability  and  mission  readiness  of  weapon  systems.  Given  that 
many  already  mature  platforms  are  programmed  into  the  active  inventory  for  several 
more  years,  it  is  important  to  provide  reliable  support  for  the  various  test  equipment 
responsible  for  weapon  system  condition  verification  and  diagnosis.  An  historical  view 
would  indicate  that  this  effort  is  certainly  dramatic  in  scope,  not  only  for  the  Air  Force, 
but  for  the  entire  Department  of  Defense,  as  the  Department  spent  approximately  $50 
billion  on  the  operation  and  support  of  Automatic  Test  Equipment  from  1980-1992 
(Ross,  2003). 

Past  procurement  processes  suggest  that  individual  aircraft  platform  System 
Program  Offices  developed,  improved  and  replaced  Automatic  Test  Equipment  on  an 
“as-required”  basis;  their  specific  managerial  actions  being  driven  in  parallel  support  of 
the  managed  weapon  system.  Previous  researchers  contend  that  the  Air  Force  did  not 
historically  have  directive  policy  on  tester  standardization  (Ross,  2003).  With  the 
passage  of  time,  however,  growing  numbers  of  platform-specific  pieces  of  Automatic 
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Test  Equipment,  coupled  with  ever-aging  aircraft,  have  led  to  significant  increases  in 
replacement  part  obsolescence  and  greater  challenges  in  long-tenn  weapon  system 
sustainment.  As  an  example,  the  F-15  and  B-52  are  projected  to  reach  a  5 1-  and  94-year 
life-cycle,  respectively,  before  retirement  in  2040.  (Griffith,  2004).  This  obsolescence 
and  a  concern  for  maintaining  longer-term  viability  of  Automatic  Test  Systems  and 
Equipment  in  support  of  aging  aircraft  provided  the  necessary  inertia  for  a  movement 
towards  the  procurement  and  development  of  consolidated,  common  test  equipment  that 
would  support  multiple  weapon  systems. 

The  first  appearance  of  senior-leader  direction  espousing  the  benefits  of  core, 
consolidated  Automatic  Test  Systems  and  Equipment  appeared  in  the  1994  Air  Force 
Policy  Directive  63-2.  Specifically,  the  directive  stated,  “Standardized  automatic  test 
systems  (ATS)  and  equipment  can  provide  efficiency  and  reduced  cost  by  minimizing  the 
proliferation  of  system-unique  test  equipment  while  ensuring  that  the  maintenance  and 
deployment  requirements  of  existing  and  developing  weapon  systems  and  equipment  are 
met.”  (USAF- 1 .  1994).  Furthermore,  the  directive  emphasizes  “Air  Force  policy  is  to 
minimize  new  ATS  development  and  to  acquire  automatic  test  systems  that  have 
applications  across  multiple  weapon  systems.”  (USAF-1,  1994). 

In  1998,  the  Air  Force  Chief  of  Staff  directed  Headquarters  Air  Force  Materiel 
Command  to  establish  a  Common  Support  Equipment  office  to  facilitate  the  combination 
of  similar  legacy  Automatic  Test  Equipment  into  common  core  test  equipment.  Still, 
despite  this  initiative  and  the  presence  of  Air  Force  policy  regarding  the  acquisition  and 
development  of  common  test  equipment,  two  different  Automatic  Test  Equipment 
management  strategies  continue  to  exist  in  practice  today.  Some  Automatic  Test 
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Equipment  continues  to  be  primarily  supported  by  individual  System  Program  Offices, 
while  other  Automatic  Test  Equipment  is  consigned  to  the  Automated  Test  Systems 
Division  at  Warner-Robins  Air  Logistics  Center  (WR-ALC/LEA)  for  sustainment 
through  the  remainder  of  the  life-cycle.  On  the  surface,  the  first  approach  sets  the  stage 
for  wasted  resources,  since  there  is  a  strong  likelihood  that  multiple  weapon  systems, 
when  encountering  similar  needs  from  field  users,  could  individually  fund  and  acquire 
identical  or  similar  testers.  (MacAulay  Brown,  2002).  The  second  approach,  however, 
introduces  a  greater  level  of  complexity  to  Automatic  Test  Equipment  management,  since 
individual  System  Program  Office  managers  would  need  to  involve  and  accommodate  the 
needs  of  other  System  Program  Offices,  as  well  as  the  future  sustainment  concerns  of  the 
Automated  Test  Systems  Division  at  Warner-Robins  Air  Logistics  Center,  throughout  the 
acquisition  management  process.  (MacAulay  Brown,  2002).  Previous  research  suggests 
that  since  support  funding  flows  through  individual  System  Program  Offices,  managers 
from  competing  organizations  would  be  concerned  about  paying  more  than  their  “fair 
share”  for  the  sake  of  commonality  during  the  acquisition  and  development  of  new 
testers.  (MacAulay  Brown,  2002). 

Consensus  opinion  suggests  that  common  Automatic  Test  Equipment  may 
leverage  economies  of  scale  and  afford  savings  in  tenns  of  life-cycle  costs,  but  the 
elimination  of  “single-point  ownership”  may  complicate  both  supportability  and  the 
managerial  abilities  of  individual  program  offices.  In  contrast,  while  individually 
procured  and  uniquely  managed  Automatic  Test  Equipment  may  be  more  easily  managed 
by  System  Program  Offices,  it  seems  exceptionally  wasteful  not  to  capitalize  on  the 
similarities  of  function  between  related  testers  across  weapon  systems.  In  short,  common 
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Automatic  Test  Equipment  may  add  complexity  to  equipment  procurement  and 
sustainment,  but  offers  potential  savings  in  cross-system  usage.  Equipment  managed  by 
individual  System  Program  Offices  may  benefit  from  focused,  tailored  procurement  and 
sustainment,  but  perhaps  at  the  cost  of  redundancy  or  wasted  resources. 

The  existence  of  two  distinct  strategies  suggests  the  question  of  comparison.  The 
choice  is  between  one  solution  that  is  easy  to  implement  but  somewhat  inefficient 
(individual  weapon  system  management),  and  another  (consolidated  management),  which 
is  more  complex  to  implement  and  maintain,  but  potentially  offers  efficiencies  in 
acquisition,  logistics,  and  sustainment.  (MacAulay  Brown,  2002).  Accordingly,  this 
research  is  designed  to  examine,  compare  and  investigate  the  existing  similarities  and 
differences  of  these  two  approaches.  Specifically,  this  research  focuses  on  the 
management  of  two  pieces  of  Automatic  Test  Equipment  for  a  mature  aircraft  system  - 
the  F-15. 

Problem  Statement 

United  States  Air  Force  decision-makers  need  to  understand  specific  costs,  benefits 
and  concerns  regarding  the  implications  of  two  alternative  strategies  for  Automatic  Test 
Equipment  sustainment:  1)  single-point  Automatic  Test  Equipment  management  under 
the  weapon  system  ownership  of  a  System  Program  Office  or  2)  the  consolidation  of 
Automatic  Test  Equipment  to  be  managed  at  an  existing  stand-alone  automated  test 
systems  support  office.  This  research  will  compare  the  existing  management  strategies 
for  two  pieces  of  Automatic  Test  Equipment  under  a  single  mature  weapon  system  (F-15) 
-  the  primary  difference  being  the  focal  point  of  management  support.  Both  pieces  of 
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Automatic  Test  Equipment  under  study  are  in  the  sustainment  phase.  One  is  primarily 
supported  by  the  Air  Force’s  Automatic  Test  Systems  product  group  manager  (Avionics 
Intermediate  Shop  (AIS),  WR-ALC/LEA),  while  the  other’s  focal  management  remains 
at  the  F-15  System  Program  Office  (Tactical  Electronic  Warfare  System  Intermediate 
Service  Station  (TISS),  WPAFB). 

Research  Question 

This  research  seeks  to  answer  the  question:  Is  F-15  automatic  test  equipment 
more  supportable  and  efficient  when  managed  by  a  separate  support  office  or  when 
managed  by  the  supported  weapon  system? 

Investigative  Questions 

Multiple  questions  will  be  addressed  in  order  to  answer  the  research  question: 

1 .  What  are  the  differences  in  how  Automatic  Test  Equipment  is  being  managed  by 
the  respective  programs? 

a.  Prior  to  making  assessments  of  supportability  and  efficiency,  the 
structural  and  practical  differences  in  management  technique  must  be 
identified  and  studied.  There  may  or  may  not  be  dissimilarity 
between  the  System  Program  Office  and  the  depot  regarding 
communication,  fonnal  requests,  deadlines,  timelines,  bureaucracy, 
etc. 

2.  How  much  sustainment  funding  is  budgeted  and  funded  in  the  Program  Objective 
Memorandum  (POM)  and  Future  Year  Defense  Plan  (FYDP)  by  each  program? 
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a.  As  an  example,  has  one  equipment  system  historically  received  a 
greater  proportion  of  its  requested  budget?  Has  the  flow  of  funding 
been  more  responsive,  timely  and  consistent?  Where  do  equipment 
Program  Objective  Memorandum  inputs  rank  among  Air  Staff  and 
Major  Command  funding  priorities? 

3.  What  are  System  Program  Office,  depot  and  Major  Command  assessments  of 
Automatic  Test  Equipment  sustainability  for  the  two  programs? 

a.  This  question  seeks  to  compare  both  the  volume  and  severity  of 
sustainability  issues  (e.g.  hardware  and  component  reparability  and 
availability)  between  the  two  systems. 

4.  What  are  Major  Command  assessments  of  field  units’  abilities  to  support  assigned 
equipment  with  resources  provided  by  the  System  Program  Office  and  depot? 

a.  For  example,  do  technicians  and  frontline  users  receive  the  parts  they 
need,  and  are  they  received  in  a  timely  fashion? 

b.  Alternatively,  how  often  are  field  unit  personnel  confronted  with 
inadequate  data  from  technical  orders  or  insufficient  troubleshooting 
support,  necessitating  workarounds  and  “creative  solutions”  to  return 
equipment  to  operational  status? 

Scope  and  Limitations  of  Research 

This  research  effort  focuses  solely  on  comparing  the  management  of  two  pieces  of 
Automatic  Test  Equipment  within  the  F-15  weapon  system.  The  selection  of  the  Tactical 
Electronic  Warfare  System  Intennediate  Service  Station  and  the  Avionics  Intermediate 
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Shop  as  the  two  representative  systems  for  this  study  was  made  on  the  recommendation 
and  concurrence  of  experts  within  the  F-15  weapon  system  and  test  equipment 
community.  While  the  Avionics  Intermediate  Shop  is  not  “common”  to  other  weapon 
systems,  its  primary  managerial  support  is  focused  at  the  Air  Force’s  Automated  Test 
Systems  Division  (WR-ALC/LEA),  and  was  deemed  by  field  experts  and  the  researcher 
as  a  representative  system  for  the  purposes  of  this  study.  Additionally,  both  of  these 
systems  pre-existed  the  implementation  of  the  Secretary  of  the  Air  Force’s  guidance  on 
new  Automatic  Test  Systems  acquisition  in  1994.  As  a  result,  both  systems  are  observed 
in  the  sustainment  phase.  The  aim  of  this  research  is  to  investigate  and  potentially 
discover  any  significant  differences  in  managerial  practice  between  depot  and  weapon 
system  (F-15)  System  Program  Office  managers.  The  resultant  data  analysis,  therefore, 
is  limited  to  the  management  of  equipment  under  study,  and  its  applicability  should  be 
carefully  extrapolated  to  other  managed  systems. 

Methodology 

The  researcher  chose  a  qualitative  research  methodology  and  specifically  a  case 
study  strategy  for  the  research  design.  Based  on  the  recommendation  of  subject  matter 
experts,  a  suitable  equipment  candidate  for  study  was  chosen  from  each  of  the  two 
management  communities  for  Automatic  Test  Equipment.  This  researcher  relied 
primarily  on  structured  interviews  and  document  analysis  to  collect  data  on  the 
management  approaches  for  the  two  pieces  of  equipment  under  study.  In  accordance 
with  case  study  data  analysis,  this  researcher  sought  to  identify  convergence  and 
triangulate  data.  (Leedy  and  Ormrod,  2001)  Additionally,  pattern  identification,  single 


17 


instance  interpretation  and  overall  data  synthesis  were  employed  as  a  means  to  draw 
conclusions  about  and  comparisons  between  the  two  management  strategies.  (Creswell, 
1998). 

Summary 

This  chapter  introduced  the  research  objectives  and  approach  contained  within 
this  study.  We  began  by  providing  a  brief  overview  of  United  States  Air  Force 
Automatic  Test  Equipment,  the  importance  and  cost  of  supporting  Automatic  Test 
Equipment,  and  the  two  managerial  concepts  currently  being  used  to  manage  Automatic 
Test  Equipment.  Next,  a  problem  statement  was  formulated  and  a  research  question  was 
identified  to  guide  the  direction  of  the  study.  Additionally,  four  investigative  questions 
were  introduced  to  further  narrow  the  scope  of  the  research  and  provide  a  framework  to 
guide  data  collection  and  analysis  efforts. 

Additionally,  limitations  affecting  the  scope  of  this  research  were  identified  and  a 
research  methodology  strategy  was  outlined.  In  the  following  chapter,  a  variety  of 
specific  literature  is  examined  addressing  the  management  of  Automatic  Test  Equipment. 


18 


II.  Literature  Review 


Background 

In  examining  the  related  literature,  the  second  chapter  familiarizes  the  reader  with 
Automatic  Test  Systems  and  Equipment  by  defining  the  system  terms  themselves,  as  well 
as  discussing  in  greater  depth  the  characterization  of  “common”  test  equipment.  Next,  a 
review  of  associated  official  policy  and  guidance  documentation  examines  both  historical 
and  current  approaches  to  the  acquisition  and  management  of  Automatic  Test  Equipment. 
Additionally,  the  current  state  of  United  States  Air  Force  Automatic  Test  Equipment  is 
discussed  using  references  of  existing  third-party  reports  and  government  audits  into 
existing  practices.  In  preparation  for  the  in-depth  examination  specific  to  this  study,  the 
researcher  will  then  provide  specific  background  on  Automatic  Test  Equipment  for  the  F- 
15  weapon  system,  including  organizational  management  structure.  The  chapter  then 
concludes  with  specific  familiarization  material  pertaining  to  the  two  test  equipment 
systems  chosen  for  this  study. 

Introduction  to  Automatic  Test  Systems  and  Equipment 

Automatic  Test  Systems  and  Equipment  are  carefully  defined  within  the  relevant 
USAF  guidance  on  their  acquisition.  Specifically,  “ATS  consist  of  ATE  and  the  Test 
Program  Sets  (TPS)  required  for  testing  the  unit-under-test  (UUT).  Test  program  sets 
include  software,  interface  devices  and  necessary  data.  ATS  are  equipment  designed  to 
conduct  analysis  of  functional  or  static  parameters  automatically  and  to  evaluate  the 
degree  of  unit-under-test  perfonnance  degradation.  They  may  be  used  to  perform  fault 
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isolation  of  unit-under-test  malfunctions.  The  decision  making,  control,  or  evaluative 
functions  are  conducted  with  minimum  reliance  on  human  intervention  and  usually  done 
under  computer  control”  (USAF-1,  1994).  In  simpler  terms,  an  Automatic  Test  System  is 
the  broadest  category;  Automatic  Test  Equipment  and  Test  Program  Set  are  nested 
categories  within,  as  illustrated  in  Figure  1 . 


Automatic  Test  System  (ATS) 


“  Automatic  Test  Equipment  (ATE)  " 
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TPS  Development  Tools 


Figure  1.  ATS,  ATE,  &  TPS  Categorization  (Fletcher,  1998). 


Automatic  Test  Equipment  includes  both  the  hardware  and  software  required  for 
system  operation,  but  not  the  software  required  to  execute  specific  tests  (this  is  part  of  a 
given  Test  Program  Set).  Hardware  size  and,  consequently,  Automatic  Test  Equipment 
portability  vary  greatly.  Some  systems  are  easily  transportable  in  hardened  suitcases; 
others  are  bookshelf-sized  equipment  racks  that  are  more  or  less  fixed  in  place.  Still, 
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observers  will  find  Automatic  Test  Equipment  employed  in  environments  hostile  and 
non-hostile,  local  and  remote.  Specialized  Automatic  Test  Equipment  may  be  ruggedly 
designed  to  withstand  harsh  climates  or  use  aboard  ships;  other  locations  may  employ 
more  generalized  commercial  off-the-shelf  (COTS)  systems.  Regardless  of  the 
specificity  of  particular  Automatic  Test  Equipment,  the  primary  functions  are  basically 
the  same.  First,  a  computer  is  used  to  control  one  or  more  test  instruments,  such  as  signal 
generators,  voltmeters  or  waveform  analyzers.  These  instruments  are  guided  by  test 
software  programs  to  stimulate  a  designated  circuit  or  electrical  component  in  the  unit 
under  test.  The  instruments  will  then  measure  electrical  response  at  the  relevant  pins, 
ports  or  connections  to  determine  the  status  of  the  unit-under-test  (e.g.  meets  or  does  not 
meet  specifications,  presence  of  one  or  more  faults).  The  specific  electronics  tested  by 
Automatic  Test  Equipment  include  “black  boxes”  (line-replaceable  units),  circuit  cards 
(shop-replaceable  units),  and  in  some  cases,  all-up  rounds  themselves.  Finally,  most 
Automatic  Test  Equipment  will  also  perform  internal  maintenance  and  diagnosis 
operations,  such  as  self-calibration  (Fletcher,  1998). 

Test  Program  Sets  most  often  are  comprised  of  specific  program  software,  interface 
hardware  (e.g.  connection  devices,  fixtures  or  cables)  and  documentation.  Test  Program 
Set  software  will  analyze  the  response  measurements  taken  by  Automatic  Test 
Equipment  test  instruments  and  relay  pertinent  information  to  the  operating  technician 
(e.g.  fault  detection,  cause  of  failure,  satisfactory  function)  (Fletcher,  1998). 

Automatic  Test  Systems  may  be  found  at  all  levels  of  maintenance:  production 
factory,  depots,  contractor  facilities,  intermediate  repair  facilities  and  organizational 
shops.  The  current  Department  of  Defense  inventory  numbers  over  400  different 
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Automatic  Test  Systems  (GAO,  2003).  As  one  might  expect,  there  are  tens  of  thousands 
of  individual  Test  Program  Sets  (Vandenberg,  2004)  to  accompany  these  systems. 

Within  the  United  States  Air  Force,  the  Wamer-Robins  Air  Logistics  Center  Automatic 
Test  Systems  Division  (WR-ALC/LEA)  manages  seventy-percent  and  Warner  Robins 
Air  Logistics  Center  as  a  whole  manages  eighty  three-percent  of  the  3 1,783  Air  Force 
managed  Automatic  Test  Systems  and  Equipment  items  (Johnson,  2004).  To  generalize, 
in  its  present  state,  there  is  negligible  interoperability  across  different  United  States  Air 
Force  Automatic  Test  Systems,  let  alone  across  the  military  services.  Test  Program  Sets 
developed  for  a  given  Automatic  Test  System  generally  cannot  be  easily  migrated  to 
another  system.  Some  suggest  that  this  lack  of  flexibility  is  severely  limiting,  and  a  key 
contributor  to  the  accumulation  of  more  and  more  test  sets  to  attain  the  desired  (or,  as  it 
were,  needed)  capacity  for  each  supported  unit-under-test  (Vandenberg,  2004). 

“Common”  Automatic  Test  Systems  Defined 

For  the  purposes  of  this  study,  the  term  commonality  is  used  to  describe  Automatic 
Test  Systems  that  can  be  used  to  test  more  than  one  type  of  weapon  system.  More 
importantly,  the  term  “common”  will  be  used  a  convenient  means  to  categorize  all 
Automatic  Test  Systems  that  are  managed  by  the  Automated  Test  Systems  Division  at 
Wamer-Robins  Air  Logistics  Center  (WR-ALC/LEA),  to  include  managed  Automatic 
Test  Equipment  that  is  not  used  by  more  than  one  weapon  system.  Although  all 
Automatic  Test  Systems  under  the  focused  management  of  WR-ALC/LEA  may  not 
possess  a  degree  of  commonality,  this  researcher  wished  to  draw  a  distinction  between  1) 
single-point  Automatic  Test  System  ownership  by  individual  System  Program  Offices 


22 


and  2)  separate  Automatic  Test  System  ownership  by  Wamer-Robins  Air  Logistics 
Center’s  Automated  Test  Systems  Division.  Furthermore,  in  terms  of  future  implications, 
any  newly  acquired  Automatic  Test  Systems  will  ultimately  fall  under  the  sustainment 
ownership  of  WR-ALC/LEA  following  traditional  commodity  consignment  processes, 
which  will  be  discussed  in  greater  detail  later  in  this  chapter.  As  such,  this  researcher 
theorized  that  an  examination  of  existing  management  practices  for  equipment  at  both 
System  Program  Office  and  depot  (WR-ALC/LEA)  levels  would  yield  beneficial 
information  for  Air  Force  decision-makers,  regardless  of  the  chosen  equipment’s  level  of 
commonality. 

Interestingly,  the  existing  literature  is  not  as  explicit  in  its  definition  of  “common” 
as  one  might  expect.  Within  published  Air  Force  guidance,  the  tenn  “standard”  is 
favored  over  “common”.  Air  Force  Policy  Directive  63-2  defines  standardized 
Automatic  Test  Systems  as  “automatic  test  systems  that  have  applications  across  multiple 
weapon  systems”  (US AF-1,  1994).  Alternatively,  Air  Force  Instruction  63-201  states 
“Standardized  ATS  are  ATS  that  can  satisfy  the  requirements  of  multiple  systems  and 
weapon  systems  and  meet  designated  architecture  and  interface  standards”  (USAF-3, 
1994).  Existing  studies  make  no  mention  of  architecture  or  interface  standards,  instead 
offering  a  more  general  definition  similar  to  that  of  Air  Force  Policy  Directive  63-2.  The 
2002  MacAulay  Brown  Report  describes  common  Automatic  Test  Systems  only  in  terms 
of  “servicing  multiple  platforms”  (MacAulay  Brown,  2002).  Additionally,  a  2003 
General  Accounting  Office  report  explains  test  equipment  commonality  as  “used  on 
multiple  airframes  and  weapon  systems”  (GAO,  2003). 


23 


Automatic  Test  Systems  and  Equipment  Policy  and  Guidance 

This  section  will  discuss  the  existing  published  guidance  on  Automatic  Test 
Systems  acquisition  and  management.  There  is  a  variety  of  published  policy  across  the 
defense  hierarchical  structure,  beginning  with  the  highest  level  in  the  form  of  Department 
of  Defense  regulations.  Next,  the  broad  guidance  of  United  States  Air  Force  publications 
Air  Force  Instruction  63-201  and  Air  Force  Policy  Directive  63-2  is  reviewed.  Finally, 
publications  from  Automatic  Test  Systems  management  organizations  at  the  Department 
of  Defense  Automatic  Test  Systems  Executive  Agent  and  at  the  Wamer-Robins  Air 
Logistics  Center  Automated  Test  Systems  Division  are  examined. 

DoD  Publications 

A  series  of  Department  of  Defense  directives  and  instructions  serves  as  the 
foundation  for  the  acquisition  of  new  systems.  These  policies  directly  affect  the  actions 
of  the  program  manager  (PM)  overseeing  a  system’s  acquisition.  Specifically,  DoDI 

5000.1,  The  Defense  Acquisition  System,  mandates  that  program  managers  consider 
supportability,  life-cycle  costs,  performance,  and  schedule  as  comparable  factors  when 
making  program  decisions.  Furthennore,  supportability  is  to  be  considered  as  a  key 
component  of  performance  and  examined  throughout  the  item  life-cycle  (DOD-1,  2003). 
At  this  point,  there  is  no  clear  direction  driving  an  Automatic  Test  Systems  program 
manager  towards  multiple  weapon-system  commonalities.  At  best,  one  could  only 
speculate  that  life-cycle  costs  would  be  reduced  through  this  pursuit.  However,  a 
stronger  link  with  reduced  cost  resonates  within  Department  of  Defense  Instruction 

5000.2,  which  provides  specific  guidance  for  considering  commercial-off-the-shelf 
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sources  of  supply.  It  states  that  Department  of  Defense  components  shall  affirmatively 
answer,  among  others,  the  question:  Does  the  acquisition  support  work  processes  that 
have  been  simplified  or  otherwise  redesigned  to  reduce  costs,  improve  effectiveness,  and 
make  maximum  use  of  commercial  off-the-shelf  technology?”  (DOD-2,  2003). 
Effectively,  this  requires  the  program  manager  to  use  those  sources  judged  to  be  the  most 
cost-effective  throughout  a  system’s  life-cycle.  In  regards  to  commonality,  Department 
of  Defense  Instruction  5000.2  states  that,  prior  to  any  acquisition  entering  the 
“Production  and  Deployment  Phase”,  it  must  demonstrate  “acceptable  interoperability” 
(DOD-2,  2003).  The  term  “interoperability”  is  not  defined  in  the  context  of  this 
instruction. 

For  clarifying  guidance  on  the  Automatic  Test  Systems  acquisition  process,  this 
researcher  turned  to  the  DoD  Executive  Directorate  for  Automatic  Test  Systems. 

Department  of  Defense  Automatic  Test  Systems  Executive  Directorate 

In  February  2004,  The  Office  of  the  Secretary  of  Defense  (Acquisition,  Technology 
and  Logistics),  Logistics  Plans  and  Programs  office  tasked  the  United  States  Navy  to 
serve  as  the  DoD's  Executive  Director  for  Automatic  Test  Systems.  Shortly  thereafter,  in 
June  2004,  the  Deputy  Assistant  Secretary  of  the  Navy  (Logistics)  tasked  the  Naval  Air 
Systems  Command  PMA260,  the  Aviation  Support  Equipment  Program  Manager,  to 
serve  as  the  Automatic  Test  Systems  Executive  Directorate.  The  organizational  structure 
of  the  Executive  Directorate  (fonnerly  known  as  Executive  Agent)  is  depicted  in  Figure 
2. 
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ATS  EA  Organization  Chart 
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Figure  2.  ATS  Executive  Directorate  Organizational  Chart  (ATS  ED,  2004). 


Essentially,  the  Automatic  Test  Systems  Executive  Directorate’s  mission  is  to 
coordinate  acquisition,  research  and  development  for  Automatic  Test  Systems  throughout 
the  Department  of  Defense.  Under  their  published  mission  statement,  the  Directorate 
cites  four  major  goals: 

1)  Minimize  the  cost  of  automatic  testing  to  DoD 

2)  Foster  interoperability  of  automatic  test  systems  across  the  Services 

3)  Reduce  logistics  footprint 

4)  Improve  the  quality  of  test  by  leveraging  embedded  and  other  diagnostic  data 

(ATS  ED,  2004). 
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The  achieve  these  goals,  the  Directorate  further  specifies  four  tasks:  1)  minimize 
unique  types  of  Automatic  Test  Systems  within  the  Department  of  Defense,  2)  define  and 
manage  the  Automatic  Test  Systems  Open  Systems  Approach,  3)  coordinate  the 
convergence  and  modernization  of  the  Automatic  Test  Systems  inventory,  and  4)  provide 
Automatic  Test  Systems  management  tools  and  lessons  learned  (ATS  ED,  2004). 

In  an  effort  to  minimize  the  number  of  unique  Automatic  Test  Systems  entering  the 
inventory,  the  Office  of  the  Secretary  of  Defense  (OSD)  put  forth  guidance  regarding 
Automatic  Test  Systems  acquisition  procedures  in  the  fonn  of  a  Joint  Memorandum  of 
Agreement.  The  Memorandum  of  Agreement  states: 

DoD  Components  shall  minimize,  to  the  maximum  extent  feasible,  the 
introduction  of  unique  types  of  ATS  into  the  DoD  inventory  by  using  designated 
ATS  Families  or  commercial  testers  that  meet  defined  critical  interfaces  to  fulfill 
acquisition  needs  for  ATS  hardware  and  software. 

(ATS  EA  MOA,  1997). 

Most  notably,  as  indicated  by  the  italicized  text  above,  the  Memorandum  of 
Agreement  does  not  forbid  the  introduction  of  unique  Automatic  Test  Systems,  but 
establishes  a  “last  resort”  strategy  regarding  its  selection.  While  the  policy  verbiage  is 
likely  intended  to  allow  for  new  technologies  and  not  “tie  hands”,  so  to  speak,  it  is  this 
researcher’s  contention  that  this  wording  (i.e.  minimize  to  the  maximum  extent  feasible) 
is  subjective  and  open  to  interpretation.  Furthermore,  the  absence  of  strong,  unwavering 
stance  may  be  somewhat  responsible  for  the  continued  proliferation  of  unique  Automatic 
Test  Systems  despite  published  policy  advocating  its  “minimization”.  Clearly,  the 
responsibility  to  implement  the  framework  outlined  in  the  Memorandum  of  Agreement 
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belongs  to  the  individual  services;  the  interpretation  of  “unique  system  minimization” 
will  be  investigated  further  in  the  discussion  of  United  States  Air  Force  Automatic  Test 
Systems  acquisition  policy  and  guidance. 

United  States  Air  Force  Publications 

Headquarters  Air  Force  policy  is  largely  a  restatement  of  the  existing  Department  of 
Defense  acquisition  guidance  and  Automatic  Test  Systems  Executive  Directorate 
Memorandum  of  Agreement  procedures,  tailored  to  the  service’s  own  organizational 
structure.  First,  Air  Force  Instruction  63-101  implements  Air  Force  Policy  Directive  63- 
1,  Acquisition  System,  and  the  5000  series  of  Department  of  Defense  Acquisition 
Directives.  This  instruction  outlines  the  responsibilities  for  the  Secretary  of  the  Air  Force 
(SAF)  and  staff,  System  Program  Directors,  Air  Force  Materiel  Command,  Air  Logistics 
Centers,  as  well  as  item  and  product  group  managers  for  any  acquisition  program  (USAF- 
3,  1994). 

Air  Force  Policy  Directive  63-2,  Automatic  Test  Systems  and  Equipment, 
established  specific  policy  for  the  management  and  acquisition  of  Air  Force  Automatic 
Test  Systems.  In  keeping  with  the  Executive  Directorate’s  published  Memorandum  of 
Agreement,  it  features  direction  to  minimize  development  of  new  Automatic  Test 
Systems  and  acquire  common  Automatic  Test  Systems.  Specifically,  it  hierarchically 
places  established  Department  of  Defense  Automatic  Test  Systems  Families  as  the  first 
alternative  to  meet  new  requirements,  followed  by  commercial-off-the-shelf  solutions.  In 
fact,  the  policy  specifies  that  weapon  system  unique  testers  will  be  the  last  choice  and 
makes  requisite  approval  for  such  deviation  to  policy  before  unique  systems  can  be 
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selected.  In  order  to  assess  compliance  with  this  policy,  the  policy  directive  provides  a 
specific  analysis  tool  as  an  attachment.  Essentially,  a  Product  Group  Manager  (PGM)  is 
to  quantify  Automatic  Test  System  proliferation  as  a  measure  of  the  percentage  of 
peculiar  Automatic  Test  System  developments  authorized  per  total  new  Automatic  Test 
System  requirements,  as  illustrated  in  Figure  3  (USAF-1,  1994:1-4). 


Figure  3.  Measuring  Compliance  With  ATS  Non-proliferation  Policy 

(USAF-1,  1994) 

The  Automatic  Test  Equipment  Product  Group  Manager  is  the  singular  point  of 
accountability  for  all  cost,  schedule  and  performance  aspects  of  Air  Force  Automatic  Test 
Equipment  and  related  sustainment.  Per  the  policy  directive,  the  Product  Group  Manager 
is  also  given  the  co-authority  to  make  Automatic  Test  System  selection  decisions  along 
with  the  individual  System  Program  Director  establishing  an  Automatic  Test  System 
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requirement.  The  current  Automatic  Test  System  Product  Group  Manager  is  located  at 
Warner-Robins  Air  Logistics  Center,  Robins  Air  Force  Base,  Georgia,  designated  as  WR- 
ALC/LEA.  This  office,  the  Automated  Test  Systems  Division,  serves  to  promote 
acquisition  and  maintenance  of  Air  Force  test  equipment  to  meet  current  and  future 
requirements.  The  Division  Chief  represents  the  United  States  Air  Force  to  the  Executive 
Directorate  and  serves  as  the  central  point  for  Air  Force  Common  Automatic  Test 
Systems  policy  and  management  (Johnson,  2004:1-6;  Vandenberg,  2004:13). 

Air  Force  Instruction  63-201,  Automatic  Test  Systems  and  Equipment  Acquisition. 
established  procedures  for  centralizing  Automatic  Test  System  management  at  the 
Product  Group  Manager.  This  instruction  required  the  Automatic  Test  System  Product 
Group  Manager  to:  1)  establish  a  database  identifying  existing  and  preferred  Automatic 
Test  Systems  throughout  the  Air  Force,  2)  consolidate  all  Air  Force  Automatic  Test 
System  requirements,  3)  track  detailed  Automatic  Test  System  capability  and  unit-under- 
test  requirements,  and  4)  develop  an  Automatic  Test  Systems  Master  Plan  to  standardize 
Automatic  Test  Systems  into  families  of  testers.  System  Program  Directors  (SPDs), 
meanwhile,  are  tasked  to  identify  Automatic  Test  System  requirements  early  in  the 
product  development  process,  provide  updates  throughout  the  evolution  of  an  system 
acquisition  project,  and  most  importantly,  budget  and  provide  funds  to  the  Automatic 
Test  System  Product  Group  Manager  (who,  ultimately,  will  acquire  the  Automatic  Test 
System)  (USAF-3,  1994:2-3).  As  stated  previously,  the  selection  of  weapon  system 
unique  testers  is  to  be  the  last  option  among  the  evaluation  of  alternative  choices  for 
Automatic  Test  Systems.  Air  Force  Instruction  63-201  establishes  the  following 
sequential  methodology  for  Automatic  Test  System  selection: 
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1.  The  Automatic  Test  System  Selection  Evaluation  Team  considers  all  User 
Automatic  Test  System  requirements  in  the  context  of  total  Air  Force  Automatic 
Test  System  requirements. 

2.  The  Department  of  Defense  inventory  of  Automatic  Test  System  Standard 
Families,  or  modifications  to  these  systems,  are  the  first  alternatives  evaluated 
for  satisfying  Automatic  Test  System  requirements. 

3 .  The  second  alternative  evaluated  is  procurement  of  existing  items  within  the 
Department  of  Defense  inventory  of  Automatic  Test  Systems  and  Equipment. 

4.  The  third  alternative  evaluated  is  a  modification  to  existing  items  within  the 
Department  of  Defense  inventory  Automatic  Test  Systems. 

5.  If  previous  solutions  do  not  cost  effectively  meet  the  user’s  needs,  the  fourth 
alternative  evaluated  is  commercial  off  the  shelf  equipment,  appropriately 
modified.  A  waiver  is  required  unless  this  equipment  is  for  depot  use  only. 

6.  If  none  of  the  above  alternatives  can  satisfy  the  requirements,  new 
development  may  be  authorized  upon  approval  of  a  waiver  request  in 
accordance  with  Department  of  Defense  policy. 

(USAF-3,  1994) 


At  this  point,  it  becomes  readily  apparent  that  sufficient  literature  and  policy  exists 
to  effectively  minimize  the  proliferation  of  weapon  system  unique  Automatic  Test 
Systems  to  all  but  the  most  necessary  and  unavoidable  cases.  Given  this  reality,  why  has 
there  been  little  movement  in  the  way  of  commonality,  and  why  have  weapon  systems 
largely  maintained  the  status  quo  of  unique  Automatic  Test  Systems?  To  investigate  this 
question,  we  examine  published  literature  regarding  the  current  state  of  affairs  within 
United  States  Air  Force  Automatic  Test  Systems. 

Current  State  of  USAF  ATS/ATE 

2003  General  Accounting  Office  (GAO)  Automatic  Test  Systems  Report 

In  March  of  2003,  the  United  States  General  Accounting  Office  published  the 
findings  of  its  investigation  of  Department  of  Defense  Automatic  Test  Systems.  The 
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motivating  factor  behind  this  study  largely  concerned  answering  the  question  outlined 
above.  Specifically,  in  regards  to  the  Air  Force,  the  office  stated  .little  evidence 
suggests  that  consideration  is  being  given  to  the  acquisition  of  equipment  that  would  have 
common  utility  for  more  than  one  weapon  system  as  Department  of  Defense  policy 
advocates.  For  procurement  of  new  weapon  systems,  the  Air  Force  is  giving  little 
consideration  to  the  use  of  a  common  tester. . .”  (GAO,  2003).  The  report  discussed 
several  findings  to  which  this  lack  of  progress  can  be  attributed.  First,  the  office 
suggested  that  the  Executive  Directorate  does  not,  in  practice,  possess  sufficient  authority 
to  enforce  existing  Department  of  Defense  policy  regarding  the  minimization  of  unique 
Automatic  Test  Systems  at  the  service-  or  office-level.  Within  the  United  States  Air 
Force,  the  “ownership”  of  modernization  planning  by  each  System  Program  Office  may 
be  largely  to  blame: 

Since  individual  aircraft  program  offices  have  been  doing  their  own  planning  for 
modernization,  the  Air  Force  has  given  little  consideration  to  having  common 
Automatic  Test  Equipment  or  testers  that  are  interoperable  with  those  of  other 
services.  Planning  for  the  Air  Force’s  latest  aircraft  acquisition,  the  F/A-22,  calls 
for  the  development  of  automatic  test  equipment  that  will  be  unique  to  that 
aircraft.” 

(GAO,  2003). 

Additionally,  since  existing  policy  pertains  to  the  acquisition  of  new  systems,  and 
the  Air  Force  has  frequently  upgraded  existing  systems,  an  effective  “loophole”  has 
emerged  to  avoid  commonality  requirements,  regardless  of  intentions.  The  report  does 
state  that  the  Automatic  Test  Systems  Division  of  WR-ALC/LEA  was  formed  to  alter  this 
trend,  but  in  the  same  fashion  as  the  Department  of  Defense  Executive  Directorate,  this 
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office  does  not  possess  practical  authority  over  Automatic  Test  System  decisions  at 
individual  System  Program  Offices.  Moreover,  they  do  not  influence  Automatic  Test 
System  funding  to  enforce  such  requirements  or  counteract  historical  behavior. 
Additionally,  the  General  Accounting  Office  determined  that  waivers  to  Automatic  Test 
System  acquisition  policy  had,  in  many  cases,  not  been  requested  from  the  Executive 
Directorate  as  required  in  the  previously  mentioned  Memorandum  of  Agreement.  When 
waivers  were  requested,  the  Executive  Directorate  could  only  recommend  disapproval, 
lacking  firm  authority  at  the  System  Program  Office  level  (GAO,  2003). 

According  to  the  General  Accounting  Office,  the  placement  of  final  authority  at  the 

individual  program  offices  is  largely  responsible  for  the  lack  of  significant  progress 

towards  Automatic  Test  System  commonality  in  the  Air  Force.  This  researcher  would 

add  that  the  existing  published  policy  does  not  help  matters,  given  its  open-ended 

verbiage  and  unequivocal  placement  of  decision-making  authority  at  the  level  of 

individual  weapon  system  program  offices.  As  an  illustration,  the  General  Accounting 

Office  identified  several  key  observations  that  seem,  on  the  surface,  to  be  quite 

contradictory  to  unique  Automatic  Test  System  minimization  efforts: 

“The  Air  Force  is  spending  over  $  15  million  for  an  interim  modernization 
of  its  intermediate  automatic  test  equipment  for  its  B-l  aircraft  while,  at  the 
same  time,  a  new  tester  is  being  developed.  If  the  Air  Force  had  taken  the 
necessary  steps  to  replace  this  obsolete  tester  in  a  timely  manner,  these 
duplicative  costs  could  likely  have  been  avoided,  and  overall  Automatic 
Test  Equipment  modernization  costs  reduced.  According  to  an  Air  Force 
official,  the  program  office  should  have  begun  the  acquisition  of  a 
replacement  tester  several  years  ago,  but  funding  was  not  available.  The 
service  is  now  considering  acquiring  a  replacement  tester  estimated  to  cost 
$190  million”. 

“Air  Force  F/A-22  program  officials  told  us  that  they  have  not  made  a 
decision  as  to  what  testers  will  be  used  to  support  this  new  aircraft,  which 
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began  development  in  1991.  The  project  office  has  not  ensured  that  all 
components  for  the  F/A-22  can  be  tested  with  a  single  tester.  Project 
officials  told  us  that  the  F/A-22  is  a  very  complex  aircraft  and  that 
opportunities  to  take  advantage  of  common  equipment  will  be  limited.  Yet, 
the  same  contractor  that  is  developing  the  F/A-22  is  also  involved  in  the 
JSF  [Joint  Strike  Fighter],  which  is  also  very  advanced  and  complex  and 
which  uses  a  common  family  of  testers.  While  current  projections  of 
Automatic  Test  Equipment  costs  are  not  available,  estimates  made  early  in 
the  F/A-22  development  phase  exceeded  $1.5  billion.” 

(GAO,  2003). 


The  General  Accounting  Office  offered  several  recommendations  in  summary  at  the 
report’s  conclusion,  including  recommendations  including:  1)  a  top-down  reemphasis  of 
existing  Department  of  Defense  acquisition  policy,  2)  reconsideration  of  the  Navy  (or  any 
uniformed  service)  for  appointment  as  Executive  Agent,  3)  granting  the  Executive  Agent 
increased  authority  and  resources  and  4)  directing  the  services  to  develop  modernization 
plans  (GAO,  2003). 

2002  Air  Force  Common  Automatic  Test  Equipment  (AFCATE)  Report 

This  study  was  completed  by  a  contractor,  MacAulay  Brown,  Inc.,  to  provide  an 
overview  of  the  current  status  of  intennediate-level  Automatic  Test  Systems,  as  well  as  to 
suggest  direction  for  future  sustainment  and  capability  development.  The  authors  of  this 
report  sought  to  provide  a  vision  for  the  Air  Force  that  will  allow  the  evolution  of 
“program  specific  automatic  test  equipment  to  more  common  standard  Automatic  Test 
System  platforms”  (MacAulay  Brown,  2002:  15).  Not  surprisingly,  they  suggest  that 
decision-makers  be  flexible  to  support  the  current  needs  of  the  warfighter,  and  mindful  of 
pressing  logistical  issues,  such  as  deployment  needs  or  concerns  that  may  impact  the 
physical  specifics  of  a  given  test  system.  More  importantly,  however,  the  authors  shed 
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light  on  a  very  important  reality  within  defense  acquisitions  and  business  strategy  in 
general  -  suboptimization.  Specifically,  the  report  states,  “it  must  be  recognized  that  the 
benefits  of  short-term  expediency  can  often  outweigh  the  long-tenn  benefits  of  rigid 
adherence  to  a  global  strategy.  Each  decision  should  be  made  in  the  dual  contexts  of  the 
roadmap  and  the  supported  weapon  system  requirements”  (MacAulay  Brown,  2002:  15). 
In  its  current  state,  program  offices  are  posed  against  each  other  in  competition  for 
sustainment  dollars  and  budget  approvals.  While  in  principle,  representatives  from 
individual  weapon  system  programs  should  collectively  make  the  best  use  of  scarce 
resources  for  the  “greater  good”,  the  reality  is  more  along  the  lines  of  “every  program  for 
itself’.  Accordingly,  one  of  the  elements  this  research  intends  to  address  is  the  historical 
funding  of  multiple  Automatic  Test  System  programs. 

After  providing  substantive  background  information  and  the  suggested  roadmap  for 
future  Automatic  Test  System  portfolio  management,  the  report  progressed  to  a  review, 
by  weapon  system,  of  the  current  state  of  existing  intermediate  level  testers.  As  this 
research  is  a  case  study  for  the  F-15  platform,  the  relevant  discussion  for  this  weapon 
system  will  be  summarized  below. 

The  F-15A/B  aircraft  declared  Initial  Operational  Capability  (IOC)  nearly  three 
decades  ago,  in  July  of  1975.  At  this  point,  the  array  of  test  equipment  included  a  set  of 
six  stations  called  the  Avionics  Intermediate  Shop  (AIS)  and  the  Tactical  Electronic 
Warfare  System  (TEWS)  Intermediate  Test  Equipment  (TITE).  In  the  early  to  mid- 
1980’s,  the  AIS  stations  had  their  processors  and  many  controllers  upgraded.  Not  long 
afterward,  the  F-15E,  known  as  the  Strike  Eagle,  was  introduced  into  operational  usage. 
In  conjunction  with  this  introduction,  the  TEWS  Intermediate  Support  System  (TISS) 
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replaced  TITE  for  all  F-15  units.  Additionally,  the  Mobile  Electronic  Test  Set  (METS) 
replaced  three  of  the  original  AIS  stations,  and  the  Aircraft  Radar  Test  Station  (ARTS) 
replaced  another.  Finally,  in  the  1990s,  ARTS  was  upgraded  to  the  Enhanced  ARTS 
(EARTS).  Initially,  field  users  had  good  success  with  this  array  of  support  equipment. 
Unfortunately,  worsening  obsolescence  and  diminishing  manufacturing  sources  are 
plaguing  the  F-15  community,  resulting  in  numerous  parts  availability  problems  today, 
particularly  with  the  AIS.  To  address  this  problem,  development  of  the  Electronic 
System  Test  Set  (ESTS)  was  begun  in  1992.  This  tester  was  intended  to  provide  support 
for  the  bulk  of  the  workload  occupied  by  AIS  and  METS.  Initial  fielding  began  in  2000; 
however  this  replacement  tester  already  encountered  obsolescence  issues  (MacAulay 
Brown,  2002:  57). 

A  snapshot  of  MacAulay  Brown’s  2002  summary  of  planned  F-15  intermediate  level 
tester  upgrades  or  modifications  actions  is  included  in  Figure  4. 
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Figure  4.  F-15  Test  Equipment  “Road-Ahead”  Timeline  (MacAulay  Brown,  2002). 

ATS  Obsolescence  and  DMSMS  Issues 

Consensus  opinion  would  suggest  that  parts  obsolescence  is  one  of  the  greatest 
problems  plaguing  Automatic  Test  Systems  today.  The  2003  General  Accounting  Office 
report  suggested  three  factors  that  contribute  to  the  continuing  problem:  first,  the  simple 
fact  of  time  and  the  increasing  Automatic  Test  Systems  age;  second,  explosive  advances 
in  technology  that  can  make  new  versions  of  Automatic  Test  Systems  obsolete  before 
reaching  the  field;  and  third,  parts  scarcity  due  to  assembly  line  closures  commercial 
manufacturing  source  elimination,  commonly  known  as  Diminishing  Manufacturing 
Sources  and  Materiel  Shortages  (DMSMS).  As  an  illustration,  the  Automated  Test 
Systems  Division  at  Warner-Robins  Air  Logistics  Center  anticipates  two-,  five-,  and  ten- 
year  projections  of  14%,  31%,  and  55%  obsolescence  rates  for  systems  under  their 
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management  (Johnson,  2004).  Within  the  Air  Force’s  airborne  electronic  warfare, 
common  avionics  and  global  positioning  system  domains,  23,550  parts  are  now  classified 
as  obsolete,  with  the  total  number  expected  to  grow  to  30,994  by  2014  (Griffith,  2004). 
The  common  practice  of  extending  weapon  system  life-cycles  raises  the  propensity  for 
obsolescence  and  scarcity,  not  only  for  the  aircraft,  but  also  for  the  supporting  elements, 
such  as  associated  Automatic  Test  Systems  (Griffith,  2004).  In  regards  to  the  rapid 
progress  of  electronic  technology,  some  suggest  that  components  developed  with  current 
technology  will  be  obsolete  within  five  years.  It  has  been  estimated  that  200,000 
commercial  electronic  items  became  obsolete  in  2003.  Clearly,  obsolescence  issues  must 
be  considered  throughout  a  product  life-cycle,  from  conceptual  idea  and  acquisition 
through  modernization  and  sustainment  practices  (Taylor,  2004). 

Additional  evidence  suggests  that  commercial  manufacturers  are  considering 

obsolescence  as  an  inevitable  truth  at  the  product  development  stage.  One  author 

comments  on  the  implications  this  has  for  equipment  users: 

“Manufacturers  are  adopting  a  ‘planned  obsolescence’  marketing  approach.  By 
introducing  newer  and  better  features  in  new  models,  the  manufacturer  can 
leapfrog  his  competitor  with  a  unique  product.  This  feature  shift  makes  it 
difficult  to  identify  an  appropriate  replacement  unit  and  makes  it  significantly 
more  costly  to  repair  or  directly  replace  with  an  item  from  the  original 
equipment  manufacturer.” 

(Ostrow,  2003). 

Complications  are  not  limited  to  the  whole  replacement  of  equipment,  but  also  the 
replacement  of  subassemblies  or  repair  components.  Once  a  repair  or  replacement 
diagnosis  has  been  made  by  users  in  the  field,  manufacturers  often  develop  a  “next- 
generation”  replacement  instrument,  which  is  often  not  form,  fit  or  function  compatible 
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with  the  original  system  (Ostrow,  2003).  In  either  case,  equipment  and  item 
obsolescence  can  and  does  result  in  exhaustive  challenges  and  difficulties  for  managers 
of  defense  test  equipment.  The  next  section  presents  specifics  regarding  the  United 
States  Air  Force  F-15  weapon  system  and  its  management  structure. 

F-15  Automatic  Test  Systems  and  Equipment 

We  provide  the  reader  with  a  brief  background  of  traditional  acquisition  and 
procurement  processes  within  the  F-15  test  equipment  community,  as  well  as  initial 
descriptions  of  the  equipment  and  offices  under  study  in  this  research. 

F-15  System  Program  Office 

The  F-15  aircraft  is  currently  flown  by  21  United  States  Air  Force  and  Air  National 
Guard  units  and  three  foreign  countries:  Israel,  Japan  and  Saudi  Arabia.  Its  management 
community  is  located  at  two  different  geographic  locations:  Wright-Patterson  Air  Force 
Base,  Ohio  and  Warner-Robins  Air  Force  Base,  Georgia.  Still,  the  System  Program 
Office  is  principally  structured  as  one  organization  with  a  single  lead  manager  -  the 
System  Program  Director  (SPD).  This  individual  serves  as  the  lead  interface  with  combat 
customers;  for  the  F-15  community,  these  are  Air  Combat  Command  (ACC),  United 
States  Air  Forces  in  Europe  (USAFE),  Pacific  Air  Forces  (PACAF),  Air  Education  and 
Training  Command  (AETC)  and  the  Air  National  Guard  (ANG).  The  concept  of 
presenting  a  single  management  interface  to  the  warfighter  is  known  as  Integrated 
Weapon  System  Management  (IWSM).  This  concept  was  initiated  when  the  Air  Force 
Systems  Command  (AFSC)  and  Air  Force  Logistics  Command  (AFLC)  integrated  under 
the  new  Air  Force  Materiel  Command  in  the  early  1990s.  Before  this  merger,  the 
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responsibilities  of  the  two  commands  could  be  summarized  as  follows:  Air  Force 
Systems  Command  was  responsible  for  initial  research,  development  and  acquisition, 
while  Air  Force  Logistics  Command  was  largely  responsible  for  logistics  support  and 
sustainment.  Under  the  Integrated  Weapon  System  Management  concept,  Air  Force 
Materiel  Command  would  be  responsible  for  all  aspects  without  division  -  commonly 
known  as  “cradle -to-grave”  weapon  system  management  (F-15  SPO-2,  2004). 

As  a  result,  the  F-15  System  Program  Director  today  has  formal  responsibility 
oversight  for  every  component  of  the  F-15  platfonn.  However,  the  historical 
management  divisions  identified  above  are  still  more  or  less  in  place  today.  Chances  are 
that  a  system  in  the  acquisition  phase  will  have  lead  management  focused  at  the  North 
Office  (Wright-Patterson),  while  components  in  sustainment  are  more  likely  to  have 
principal  oversight  located  at  the  South  Office  (Wamer-Robins).  Clearly,  the  intention  of 
the  Integrated  Weapon  System  Management  concept  and  the  F-15  System  Program 
Office  is  to  create  a  shared  ownership  at  both  locations  (F-15  SPO-2,  2004). 

Commodity  Class  Consignment 

The  F-15  System  Program  Office  clearly  defines  Commodity  Class  Consignment 

(C  )  within  its  own  operating  instruction  63-103.  It  is  defined  as  follows: 

The  process  of  transitioning  systems/functions  from  the  developing  organization 
to  the  supporting  organization.  While  a  system  may  be  entrusted  to  the  care  of 
the  supporting  organization,  an  avenue  exists  for  the  supporting  organization  to 
receive  technical  and  management  support  from  the  developing  organization. 

C  establishes  a  partnership  between  the  F-15  SPO  and  supply  chain 
management  organizations. 

(F-15  SPO- 1,2002) 
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Commodity  Class  Consignment  replaced  in  nomenclature  the  process  of  Program 
Management  Responsibility  Transfer  (PMRT).  Essentially,  this  process  marks  the 
handover  of  day-to-day  management  for  a  given  commodity  (line-replaceable  unit, 
support  equipment,  etc.)  from  one  group  to  another.  The  first  group,  the  developing 
organization,  is  responsible  for  a  system’s  acquisition,  design  and  implementation.  The 
second  group,  the  supporting  organization,  is  responsible  for  the  sustainment  and 
management  of  that  system  after  initial  development  and  fielding  (Robinson,  2004). 
Within  the  F-15  community,  internal  consignment  is  defined  as  the  transition  from  the 
Aeronautical  Systems  Center  at  Wright-Patterson  AFB  (ASC/FBA)  to  the  South  F-15 
System  Program  office  at  Warner-Robins  (WR-ALC/LF).  External  consignment  is 
defined  as  the  transition  from  ASC/FBA  to  a  supply  chain  manager  at  one  of  the  Air 
Force’s  Air  Logistics  Centers. 

F-15  Automatic  Test  Systems  are  subject  to  the  same  requirements  as  other  systems 
eventually  bound  for  consignment.  Next,  we  provide  a  brief  introduction  to  the  two  test 
equipment  systems  that  were  examined  within  this  case  study. 

Tactical  Electronic  Warfare  System  Intermediate  Service  Station  (TISS) 

TISS  performs  intermediate  level  maintenance  support  for  all  of  the  F-15  electronic 
warfare  line-replaceable  units.  Currently  a  six -bay  system  as  illustrated  in  Figure  5,  the 
execution  of  an  ongoing  technology  upgrade  will  reduce  the  station’s  footprint  to  three 
bays  (Ellis,  2004). 
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Figure  5.  TISS  Station 


(Ellis,  2004) 


The  TISS  management  and  support  staff  is  distributed  at  both  the  F-15  System 
Program  Office  North  (Wright-Patterson),  and  at  Warner-Robins  Air  Logistics  Center 
(WR-ALC).  The  relevant  personnel  at  the  North  office  support  issues  relating  to 
engineering,  logistics,  technical  orders,  configuration  and  data.  Additionally,  the  lead 
Major  Command,  Air  Combat  Command,  has  a  liaison  on  staff  at  the  North  System 
Program  Office  (Wright-Patterson).  WR-ALC/LSRAC  is  responsible  for  the 
management  of  associated  electronic  warfare  line-replaceable  units  that  are  tested  by 
TISS.  Finally,  TISS  spares  and  repairs  are  overseen  by  WR-ALC/LEACC.  Generally 
speaking,  the  F-15  North  System  Program  Office  has  overall  management  of  the  TISS 
program  to  include  funding  all  modifications  (software  and  hardware),  technical  orders, 
field  engineering  support  and  system  configuration  management.  The  TISS  staff  at  the 
Automated  Test  Systems  Division  at  WR-ALC  (WR-ALC/LEACC)  is  responsible  for 
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funding  and  management  of  an  ongoing  repair  contract  with  a  manufacturer,  as  well  as 
resolving  obsolescence  issues  that  do  not  affect  the  form,  fit  or  function  of  the  TISS,  such 
as  replacing  an  obsolete  component  on  a  circuit  card  assembly. 

Avionics  Intermediate  Shop  (AIS) 

The  F-15  Avionics  Intermediate  Shop  is  a  complement  of  eight  stations  that  are 
used  to  test  and  fault  line-replaceable  units  that  monitor  airframe,  engine,  navigation, 
combat  and  pilot  safety  systems.  The  complement  consists  of  the  following:  Antenna 
Test  Station,  Indicators  and  Controls,  Communication,  Navigation  and  Identification, 
Computer  Test  Station,  Display  Test  Station,  Microwave  Test  Station,  Enhanced  Aircraft 
Radar  Test  Station  and  Mobile  Electronics  Test  Set.  The  Enhanced  Aircraft  Radar  Test 
Station  and  Mobile  Electronics  Test  Set  are  used  to  test  selected  line-replaceable  units  on 
F-15E  model  aircraft  only.  The  Enhanced  Aircraft  Radar  Test  Station  replaced  the 
Antenna  Test  Station  for  the  F-15E,  while  the  Mobile  Electronics  Test  Set  replaced  the 
Computer  Test  Station,  Indicators  and  Controls,  and  Communication,  Navigation  and 
Identification  for  the  F-15E.  The  typical  layout  of  the  AIS  is  illustrated  in  Figure  6. 
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Figure  6.  Typical  AIS  Layout 

(Mathis,  2004) 

Unlike  TISS,  the  AIS  is  entirely  managed  at  the  Automated  Test  Systems  Division 
at  Warner-Robins  Air  Logistics  Center.  The  support  staff  consists  of  the  Integrated 
Product  Team:  the  logistics  manager,  item  manager,  equipment  specialist,  production 
manager,  engineer  and  contracting  officer.  Together,  this  staff  manages  the  AIS 
program. 
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Summary 


This  chapter  examined  the  related  literature  in  order  to  establish  familiarity  with  the 
definition  and  scope  of  Automatic  Test  Systems  and  Equipment.  The  component 
interplay  between  Automatic  Test  Systems  and  Test  Program  Sets  was  discussed,  and  the 
existing  guidance  for  both  the  Department  of  Defense  and  the  United  States  Air  Force 
was  reviewed.  Next,  background  was  provided  on  the  current  state  of  Automatic  Test 
Systems  and  Equipment  inventory  and  management,  including  shortfalls  and  challenges 
facing  the  test  equipment  community.  Finally,  the  F-15  weapon  system  management 
structure  was  introduced,  as  well  as  the  specific  equipment  programs  under  study  within 
this  research  effort. 

Next,  chapter  three  discusses  the  selection  and  evolution  of  a  research  methodology 
for  this  study. 
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III.  Methodology 


Chapter  Overview 

The  purpose  of  this  chapter  is  to  describe  the  chosen  methodology  for  organizing 
the  research  design  and  collecting  relevant  data.  In  regards  to  organizing  the  study,  this 
researcher  reviewed  literature  from  Creswell  (1998,  2003),  Leedy  et  al  (2001)  and  Yin 
(2003),  and  selected  a  qualitative  approach  with  a  case  study  as  the  specific  tradition  of 
inquiry.  Using  guidance  from  Yin  (2003),  a  series  of  protocol  questions  were  constructed 
which  served  as  the  foundation  for  targeted  questionnaires.  The  resultant  interviews 
generated  from  these  questionnaires  served  as  the  primary  source  of  data  collection. 

Once  the  data  were  gathered,  a  variety  of  case  study  analytical  tools  described  by  Yin 
(2003)  and  Creswell  (1998)  were  employed,  including  pattern-establishment,  categorical 
aggregation  and  direct  interpretation.  These  strategies  were  used  to  draw  the  conclusions 
outlined  in  chapters  four  and  five. 

At  this  point,  it  should  be  noted  that  this  research  was  conducted  in  parallel  with  a 
similar  study  (Ford,  2005).  Given  that  there  are  numerous  Automated  Test  Systems 
within  the  United  States  Air  Force  and  Department  of  Defense  inventories,  we  sought  to 
develop  their  methodological  framework  as  a  foundation  template  for  future  inquiries 
into  other  Automated  Test  Systems.  As  a  direct  consequence,  the  researchers  structured 
their  studies  in  parallel;  the  only  difference  being  the  associated  ATS  under  study  (i.e.  F- 
15  Automated  Test  Systems  vs.  Strategic  Missile  Automated  Test  Systems).  Specifically, 
the  investigative  questions,  protocol  questions  and  interview  questionnaires  were 
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designed  to  be  identical  across  these  parallel  studies.  Both  researchers  contend  that  this 
standardization  will  afford  valuable  opportunity  for  future  cross-comparison  and  serve  as 
a  baseline  for  further  analyses.  In  fact,  future  researchers  may  choose  to  investigate 
additional  equipment  management  strategies  using  this  existing  framework  for  data 
collection.  Specific  commentary  on  future  research  possibilities  is  provided  in  chapter 
five.  This  being  said,  the  next  section  discusses  the  process  of  research  design. 

Research  Design 

Creswell  identifies  three  distinct  approaches  to  research  design:  quantitative, 
qualitative,  and  mixed  method  (Creswell,  2003:17).  He  defines  quantitative  research  as: 


...One  in  which  the  investigator  primarily  uses  postpositivist  claims  for 
developing  knowledge  (i.e.  cause  and  effect  thinking,  reduction  to  specific 
variables  and  hypotheses  and  questions,  use  of  measurement  and  observation, 
and  the  test  of  theories),  employs  strategies  of  inquiry  such  as  experiments 
and  surveys,  and  collects  data  on  predetennined  instruments  that  yield 
statistical  data  (Creswell,  2003:18). 


In  contrast  to  quantitative  research,  he  defines  qualitative  research  as: 


An  inquiry  process  of  understanding  based  on  distinct  methodological 
traditions  of  inquiry  that  explore  a  social  or  human  problem.  The  researcher 
builds  a  complex,  holistic  picture,  analyzes  words,  reports  detailed  views  of 
informants,  and  conducts  the  study  in  a  natural  setting  (Creswell,  1998:15). 

In  similar  fashion,  Leedy  et  al  (2001)  describe  qualitative  methods  in  terms  of  being 

“used  to  answer  questions  about  the  complex  nature  of  phenomena,  often  with  the 

purpose  of  describing  and  understanding  the  phenomena  from  the  participants’  point  of 

view”  (Leedy  and  Ormrod,  2001:101). 
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Finally,  Creswell  identifies  the  mixed  method  approach  to  research  design  that 


includes  elements  of  both  qualitative  and  quantitative  methodologies,  defined  as: 


. .  .One  in  which  the  researcher  tends  to  base  knowledge  claims  on  pragmatic 
grounds  (e.g.  consequence-oriented,  problem-centered,  and  pluralistic).  It 
employs  strategies  of  inquiry  that  involve  collecting  data  either 
simultaneously  or  sequentially  to  best  understand  research  problems.  The  data 
collection  also  involves  gathering  both  numeric  information  (e.g.  on 
instruments)  as  well  as  text  information  (e.g.  on  interviews)  so  that  the  final 
database  represents  both  quantitative  and  qualitative  information  (Creswell, 
2003:18-19). 

To  select  an  appropriate  research  approach  for  a  given  study,  one  should  be  able  to 
logically  and  clearly  define  the  relationship  between  the  problem  and  a  particular 
research  approach.  As  an  example,  a  researcher  seeking  to  identify  or  explain  the 
relationship  between  measured  variables  (e.g.  numerical  quantities)  would  likely  be  best 
suited  to  select  a  quantitative  approach  (Leedy  and  Ormrod,  2001).  In  contrast,  if  the 
researcher’s  aim  is  to  explore  and  characterize  existing  phenomena,  or  to  develop  a 
theory  regarding  such  phenomena,  a  qualitative  method  would  likely  be  the  better  choice 
(Leedy  and  Ormrod,  2001).  In  summary,  Table  1  identifies  the  three  types  of  approaches 
available  to  researchers. 


Table  1.  Research  Method  Procedures 


Quantitative 

Qualitative 

Mixed  Methods 

-  Predetermined 

-  Instrument  based  questions 

-  Performance  data,  attitude 
data,  observational  data, 
and  census  data 

-  Statistical  analysis 

-  Emerging  methods 

-  Open-ended  questions 

-  Interview  data,  observation 
data,  document  data,  and 
audiovisual  data 

-  Text  and  image  analysis 

-  Both  predetermined  and 
emerging  methods 

-  Both  open-  and  closed- 
ended  questions 

-  Multiple  forms  of  data 
drawing  on  all  possibilities 

-  Statistical  and  text  analysis 

(Creswell,  2003:17) 
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As  a  final  note  on  methodology  selection,  perhaps  the  strongest  indicators  are  the 
data  themselves.  Leedy  et  al  (2001)  clarify:  “Data  and  methodology  are  inextricably 
interdependent. .  .the  methodology  to  be  used  for  a  particular  research  problem  must 
always  take  into  account  the  nature  of  the  data”  (Leedy  and  Ormrod,  2001 : 100).  As  a 
consequence,  this  researcher  was  careful  to  consider  the  motivation  behind  the  study,  the 
questions  to  be  answered,  and  what  data  would  be  required  to  draw  conclusions. 
Moreover,  the  same  set  of  data,  when  analyzed  with  varying  methodologies,  might 
suggest  quite  different  conclusions,  each  of  which  may  vary  both  from  each  other  and, 
more  importantly,  from  the  truth. 

Qualitative  Method 

This  section  seeks  to  identity  the  distinguishing  characteristics  that  separate 
qualitative  research  from  the  other  approaches.  First,  whereas  quantitative  studies  are 
often  conducted  with  existing  data  at  an  enclosed,  defined  location  (e.g.  laboratory), 
qualitative  studies  are  often  executed  in  “the  field”;  that  is,  within  the  existing 
surroundings  of  the  subject  of  study.  Here,  the  researcher  can  gather  extensive  data  and 
valuable  insight  due  to  her  or  his  direct,  active  involvement  with  the  experiences  of  the 
subject(s)  (Creswell,  1998:16;  2003:181).  As  with  any  research,  the  researcher’s 
personal  involvement  in  conducting  the  study  makes  very  real  the  possibility  of 
introducing  bias.  Indeed,  Leedy  et  al  (2001)  suggest  that  “in  the  research  environment, 
the  researcher  cannot  avoid  having  data  contaminated  by  bias  of  one  sort  or  another” 
(Leedy  and  Ormrod,  2001 :222).  Still,  all  is  not  lost,  as  bias  is  expected  in  research, 
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provided  the  researcher  acknowledges  the  likelihood  of  biased  data  or  the  specific 
possibilities  of  bias  within  the  study  (Leedy  and  Ormrod,  2001:222  to  223). 

A  second  feature  of  qualitative  research  is  found  within  the  role  of  the  researcher. 
Within  this  approach,  the  researcher  serves  as  the  “key  instrument  of  data  collection”,  as 
the  “bulk  of  the  data  is  dependent  on  their  personal  involvement”  (Creswell,  1998:16; 
Leedy  and  Ormrod,  2001 : 102).  As  a  result,  throughout  the  study,  from  data  collection  to 
analysis,  the  researcher  must  carefully  monitor  personal  biases  and  behaviors,  making 
every  effort  to  prevent  any  negative  influence  on  the  overall  study  (Creswell  2003:184). 
Furthermore,  since  conclusions  drawn  from  qualitative  research  are  dependent  on  the 
researcher’s  interpretations  of  the  collected  data,  Creswell  (2003)  suggests  identifying 
possible  biases  at  the  earliest  opportunity,  and  rigorously  using  multiple  validity 
strategies  to  establish  credibility  and  confidence  in  the  research  findings  (Creswell, 
2003:184). 

Third,  the  researcher  may  employ  one  or  more  inquiry  strategies  as  a  procedural 
guide  for  conducting  a  qualitative  study  (Creswell,  2003:183).  Example  inquiry 
strategies  identified  by  Creswell  are:  narrative,  phenomenology,  ethnography,  case  study, 
and  grounded  theory  (Creswell,  2003:183).  In  purpose,  these  strategies  are  intended  to 
aid  the  researcher  and  provide  focus  on  data  collection,  data  analysis,  and  structured 
writing  (Creswell,  2003: 183).  Still,  the  inquiry  strategies  are  not  sets  of  step-by-step 
instructions.  Leedy  et  al  (2001)  reemphasize  this  point,  “There  are  no  magic  formulas, 
no  cookbook  recipes  for  conducting  a  qualitative  study”  (Leedy  and  Onnrod,  2001:149). 
He  further  states  that  books  written  about  qualitative  research  offer  “general  guidelines 
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based  on  the  experiences  of  those  qualitative  researchers”  and  the  specific  methods  used 
are  only  constrained  by  the  researcher’s  imagination  (Leedy  and  Ormrod,  2001:149). 

Rationale  for  Selecting  a  Qualitative  Methodology 

For  this  study,  the  primary  motivation  for  selecting  a  qualitative  methodology  was 
the  particular  nature  of  the  research  questions  and  type  of  data  being  collected.  As  a 
review,  this  study  began  by  identifying  an  overarching  research  question  followed  by 
four  supporting  investigative  questions,  each  possessing  associated  sub-questions: 

Research  Question 

This  research  seeks  to  answer  the  question:  Is  F-15  automatic  test 

equipment  more  supportable  and  efficient  when  managed  by  a  separate  support 

office  or  when  managed  by  the  supported  weapon  system? 

Investigative  Questions 

1 .  What  are  the  differences  in  how  Automated  Test  Systems  are  being  managed 
by  the  two  programs? 

2.  How  much  sustainment  funding  is  budgeted/funded  in  the  Program  Objective 
Memorandum  (POM)  and  Future  Year  Defense  Plan  (FYDP)  for  each? 

3.  What  are  System  Program  Office,  depot  and  Major  Command  assessments  of 
Automated  Test  System  sustainability  for  the  two  programs? 

4.  What  are  Major  Command  assessments  of  field  units’  abilities  to  support 
assigned  equipment  with  resources  provided  by  the  System  Program  Office 
and  depot? 
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Based  on  Creswell’s  (2003)  guidance,  the  above  research  questions  meet  the 
description  of  qualitative  research  questions  due  to  their  structure:  a  central  question 
followed  by  associated  sub-questions  (Creswell,  2003: 105).  Central  questions  are  often 
designed  to  be  broad  in  form  “so  as  to  not  limit  the  inquiry”  (Creswell,  2003: 105).  Sub¬ 
questions  are  then  employed  to  narrow  the  researcher’s  focus  and  identify  the  infonnation 
needed  to  conduct  analysis  (Creswell,  1998:101;  2003:106). 

Once  the  decision  was  made  to  conduct  this  study  using  a  qualitative  methodology, 
it  was  necessary  to  select  an  appropriate  corresponding  tradition  of  inquiry  (Creswell, 
1998:21).  Again,  the  selection  of  an  appropriate  tradition  of  inquiry  is  very  important  for 
any  study,  as  it  lays  the  groundwork  for  data  collection,  data  analysis,  and  the  writing  of 
the  research  report  (Creswell,  1998:  37;  2003:183).  Within  the  qualitative  research 
approach,  the  five  traditions  of  inquiry  are  ethnography,  grounded  theory,  case  study, 
phenomenological  research,  and  narrative  research  (also  known  as  bibliography  from  his 
previous  work)  (Creswell,  1998:7;  2003:15).  Each  tradition  is  defined  according  to  the 
descriptions  below: 

-  Ethnography:  the  study  of  an  intact  cultural  or  social  group  (or  an  individual  or 
individuals  within  the  group)  based  primarily  on  observations  and  a  prolonged 
period  of  time  spent  by  the  researcher  in  the  field  (Creswell,  1998:246) 

-  Grounded  Theory:  the  researcher  generates  an  abstract  analytical  schema  of  a 
phenomenon,  a  theory  that  explains  some  process,  action,  or  interaction 
grounded  in  the  views  of  participants  in  a  study  (Creswell,  1998:241;  2003:14) 

-  Case  Study:  the  researcher  explores  in  depth  a  program,  an  event,  an  activity,  a 
process,  or  one  or  more  individuals.  The  case(s)  are  bounded  by  time  and 
activity,  and  researchers  collect  detailed  information  using  a  variety  of  data 
collection  procedures  over  a  sustained  period  of  time  (Stake,  1995:2;  Creswell, 
1998:249;  2003:15) 
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-  Phenomenological  Research:  the  researcher  identifies  the  “essence”  of  human 
experiences  concerning  a  phenomenon,  as  described  by  the  participants  in  a 
study  (Creswell,  1998:236;  2003:15) 

Narrative  Research:  a  fonn  of  inquiry  in  which  the  researcher  studies  the  lives  of 
individuals  and  asks  one  or  more  individuals  to  provide  stories  about  their  lives 
(Creswell,  2003:15) 


Additionally,  Creswell’s  (1998)  diagram,  illustrated  in  Figure  7,  is  a  useful  tool  in 
helping  to  clarity  both  the  relationship  and  distinction  between  each  of  the  traditions 
(Creswell,  1998:37). 


Figure  7.  Differentiating  Traditions  by  Foci  (Creswell,  1998:37) 


After  reviewing  the  related  literature  for  qualitative  studies  (Creswell,  1998,  2003; 
Leedy  et  al,  2001;  Yin,  2003),  this  researcher  selected  the  case  study  strategy  as  the 
tradition  of  inquiry  for  this  research.  Creswell  (1998)  suggests  that  a  case  study  is  an 
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“exploration  of  a  ‘bounded  system’”  (Creswell,  1998:61).  More  specifically,  he  defines 
the  bounded  system  as  the  case  under  study,  and  notes  that  “several  programs  or  a  single 
program  might  be  selected  for  study”  (Creswell,  1998:61).  Given  that  the  aim  of  this 
research  effort  is  to  study  the  management  approaches  of  two  test  equipment  programs, 
the  case  study  tradition  seems  a  logical  and  sound  selection.  It  should  be  noted  that  the 
relevance  and  potential  applicability  of  the  Grounded  Theory  tradition  was  considered, 
since  one  of  the  goals  of  this  research  was  to  explore  and  explain  the  phenomena  of  two 
distinct  managerial  approaches.  However,  as  this  work  is  an  initial  study  built  to  serve  as 
a  foundation  for  future  inquiries,  it  was  determined  that  the  broader  scope  of  a  case  study 
analysis  would  be  a  more  appropriate  choice. 

Case  Study  Approach 

After  selecting  the  case  study  approach  as  the  guiding  tradition  of  inquiry,  we 
examine  in  greater  depth  the  characteristics  of  this  particular  method.  The  choice  for 
seeking  specific  case  study  guidance  from  Yin  was  made  due  to  his  wide  recognition  as 
one  of  the  foremost  authorities  on  case  study  research.  The  selection  of  the  case  study 
approach  was  reinforced  after  reviewing  the  three  conditions  outlined  by  Yin  (2003)  in 
determining  an  appropriate  research  strategy.  The  three  conditions  are:  1)  the  types  of 
research  questions,  2)  the  extent  of  control  an  investigator  has  over  actual  behavioral 
events  and  3)  the  degree  of  focus  on  contemporary  as  opposed  to  historical  events  (Yin, 
2003:5-7).  Table  2  outlines  the  recommended  research  strategy  based  on  these 
conditions. 


54 


Table  2.  Different  Research  Strategies 


Strategy 

Form  of 

Research  Question 

Requires  Control  of 
Behavioral  Events? 

Focuses  on 

Contemporary  events? 

Experiment 

How,  why? 

Yes 

Yes 

Survey 

Who,  what,  where, 
how  many,  how 
much? 

No 

Yes 

Archival 

Analysis 

Who,  what,  where, 
how  many,  how 
much? 

No 

Yes/No 

History 

How,  why? 

No 

No 

Case  Study 

How,  why? 

No 

Yes 

(Yin,  2003:5) 


The  research  and  investigative  questions  stated  in  chapter  one  consist  of  questions 
characterized  by  “how”  and  “what”  interrogations.  Although  Yin  (2003)  highlights  that  a 
case  study  strategy  should  feature  research  questions  of  the  form  “how”  and  “why”,  he 
notes  that  “what”  type  questions  can  also  be  used,  primarily  because  they  can  be 
considered  exploratory  in  nature  (Yin,  2003:5  to  6).  In  addition  to  the  types  of  questions 
being  asked  and  the  qualitative  nature  of  this  study,  the  control  of  behavioral  events  is  not 
required  -  the  purpose  of  this  study  is  to  describe  and  explore.  Furthennore,  this  study 
does  focus  on  contemporary  events;  specifically  the  current  management  strategies 
employed  by  two  different  test  equipment  programs. 

Once  the  case  study  methodology  was  selected,  this  researcher  began  to  structure 
the  specific  research  design.  Yin  (2003)  outlines  five  components  that  are  important  in 
this  regard.  The  five  components  are:  1)  a  study’s  questions,  2)  its  propositions  (if  any), 
3)  its  unit(s)  of  analysis,  4)  the  logic  linking  the  data  to  the  propositions  and  5)  the  criteria 
for  interpreting  the  findings  (Yin,  2003:21).  The  research  and  investigative  questions 
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mentioned  above  and,  more  specifically,  the  protocol  questions  and  sub-questions 
addressed  later  in  this  chapter,  satisfy  the  first  component  of  the  research  design. 

The  second  component,  study  propositions,  is  used  to  direct  the  researcher  towards 
more  focused  areas  of  study  that  should  be  examined  as  part  of  the  overall  research  effort 
(Yin,  2003:22).  In  tenns  of  defining  the  study’s  focus,  the  investigative  questions 
outlined  in  chapter  one  perfonn  this  function.  An  additional  benefit  of  using  propositions 
is  that  they  help  guide  the  researcher  to  appropriate  sources  of  data  or  evidence,  which 
are  crucial  to  developing  a  case  (Yin,  2003:22). 

In  addressing  the  third  component,  the  unit  of  analysis  defines  the  particular  case 
being  studied  (Yin,  2003:22).  The  case  or  unit  of  analysis  could  consist  of  an  individual 
(single  case  study)  or  group  of  individuals  (multiple-case  study)  (Yin,  2003:22  to  23). 
Likewise,  the  same  analogy  could  be  applied  to  studying  an  event  or  multiple  events 
(Yin,  2003:23).  For  the  purposes  of  this  study,  we  define  the  unit  of  analysis  as  the 
management  approach  to  a  specific  automated  test  system  within  the  F-15  weapon  system 
community.  These  pieces  of  equipment  were  introduced  in  chapter  two. 

According  to  Yin  (2003),  the  fourth  and  fifth  components  are  “the  least  well 
developed  in  case  studies”  (Yin,  2003:26).  This  contention  may  be  attributed  to  the 
possibility  of  researcher-induced  bias  or  subjectivity.  He  further  states  that  “these 
components  represent  the  data  analysis  steps  in  case  study  research,  and  a  research  design 
should  lay  the  foundations  for  this  analysis”  (Yin,  2003:26).  Accordingly,  a  variety  of 
commonly  accepted  analysis  strategies  were  employed,  including  pattern-matching  and 
explanation-building  (Yin,  2003:106-1 10).  Creswell  (1998)  states  that  case  study 
analysis  “consists  of  making  a  detailed  description  of  the  case  and  its  setting”  (Creswell, 
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1998:  153).  Furthermore,  Creswell  (1998)  and  Stake  (1995)  describe  four  forms  of  data 
analysis  and  interpretation  that  may  be  used  in  case  study  research  (Creswell,  1998:  153) 
These  forms  are:  categorical  aggregation,  direct  interpretation,  pattern  identification  and 
naturalistic  generalizations  (Creswell,  1998:154).  As  a  vehicle  for  these  particular  tools 
of  data  analysis,  the  researcher,  in  partnership  with  the  fellow  researcher  conducting  a 
parallel  study  (Ford,  2005)  developed  a  data  categorization  and  evaluation  matrix.  An 
example  of  this  matrix  is  shown  in  Figure  8. 
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Figure  8.  Data  Categorization  and  Evaluation  Matrix 

(Ford  &  Howe,  2005). 
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Case  Study  Design 

After  laying  out  the  research  design,  the  next  task  was  to  choose  a  case  study 
design.  The  four  types  of  designs  are  single-case  (holistic),  single-case  (embedded), 
multiple-case  (holistic),  and  multiple-case  (embedded)  (Yin,  2003:39).  A  multiple-case 
(holistic)  design  was  chosen  based  on  the  researcher’s  decision  to  compare  the 
management  strategies  for  two  systems,  AIS  and  TISS. 

Case  Study  Protocol 

Since  this  research  will  be  using  a  multiple-case  study  design,  it  is  important  to 
address  developing  the  case  study  protocol  for  this  research.  Yin  (2003)  states  “a  case 
study  protocol  is  desirable  under  all  circumstances,  but  it  is  essential  if  you  are  doing  a 
multiple-case  study”  (Yin,  2003:67).  The  case  study  protocol  helps  guide  the  researcher 
throughout  the  data  collection  and  analysis  process,  in  addition  to  increasing  research 
reliability  (Yin,  2003:67).  Yin  outlines  four  areas  that  case  study  protocol  needs  to 
address  (Yin,  2003:68): 

1 .  Introduction  to  the  case  study  and  purpose  of  protocol 

2.  Data  collection  procedures 

3.  Outline  of  case  study  report 

4.  Case  study  questions 

In  order  to  properly  prepare  for  formal  data  collection,  the  researcher  developed  a 
specific  protocol  for  each  of  the  investigative  questions.  These  protocols  can  be  found  in 
their  entirety  in  Appendices  A,  B,  C  and  D.  Next,  we  will  discuss  the  design  of  each 
investigative  question’s  research  protocol  in  greater  detail. 
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Protocol  Development:  Investigative  Questions  and  Sub-questions 
Investigative  Question  One 

The  aim  of  the  first  investigative  question  is  to  identify  any  primary  differences  in 
actual  management  practices;  that  is,  personal  behaviors  or  actions  in  the  conduct  of  day- 
to-day  business.  These  practices  may  involve  directly  or  indirectly  supporting  critical 
needs  issues  from  test  equipment  users  in  the  field,  or  they  may  be  actions  of  a  “behind 
the  scenes”  nature,  such  as  planning  for  future  concerns,  organizing  data  or  preparing 
reports.  This  researcher  chose  the  protocol  sub-questions  in  Appendix  A  to  capture 
differences  in  individual  behavior  regarding  typical  managerial  actions.  As  an  example, 
how  broad  is  the  communication  network  for  personnel  involved  in  the  support  of  this 
piece  of  test  equipment?  Is  one  management  staff  involving  different  people  in  the 
decision-making  process?  Sub-questions  one  through  four  are  focused  on  finding 
behavioral  similarities  and  differences  between  the  two  management  teams  for  TISS  and 
AIS. 

After  conducting  the  literature  review  for  this  research  effort,  it  became  readily 
apparent  that  equipment  and  parts  obsolescence  is  a  major  obstacle  facing  the  test 
equipment  community  today.  As  a  result,  specific  sub-questions  were  included  regarding 
the  impact  of  obsolescence  on  the  two  programs.  Moreover,  this  researcher  sought  to 
identify  any  differences  in  approach  as  to  how  the  management  teams  overcame  or 
otherwise  sought  to  counteract  the  effects  of  obsolescence. 

As  a  final  element  of  this  investigative  question’s  protocol,  this  researcher  included 
sub-questions  addressing  the  managers’  level  of  familiarity  with  Department  of  Defense 
policy  regarding  the  acquisition  and  development  of  common  test  equipment.  Granted, 
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the  testers  involved  in  this  study  are  neither  in  the  acquisition  phase  nor  common,  but  the 
researcher  theorized  it  might  be  useful  to  capture  the  perspectives  of  various  test 
equipment  community  members  on  the  subject  of  common  Automated  Test  Systems. 
After  all,  the  practical  evaluation  of  Automated  Test  Systems  management  is  the 
overarching  goal  of  this  research  effort.  That  being  said,  this  researcher  wished  to  take 
into  account  the  considerations  and  viewpoints  of  current  test  equipment  managers. 
Investigative  Question  Two 

The  design  of  investigative  question  two’s  case  study  protocol  can  largely  be 
characterized  as  “the  objective  of  investigative  question  one  in  tenns  of  funding”.  The 
research  question  for  this  study  addresses  the  comparison  of  test  equipment  management 
strategies  in  tenns  of  “supportability”  and  “efficiency”.  The  protocol  for  investigative 
question  two  is  designed  to  address  the  “efficiency”  portion.  It  should  be  noted  that  this 
protocol  is  not  founded  on  the  assumption  that  more  dollars  will  always  result  in  greater 
efficiency;  rather,  any  effort  to  determine  efficient  use  of  funds  must  also  incorporate  the 
supportability  aspects  targeted  by  investigative  questions  one,  three  and  four.  Still,  in  the 
competitive  world  of  defense  program  funding,  the  researcher  felt  it  necessary  to 
compare  the  two  test  equipment  programs  in  terms  of  historical  and  current  successes  in 
tenns  of  sustainment  dollars.  Specifically,  has  one  program  historically  received  a 
greater  portion  of  its  requested  funds?  If  so,  why? 

Investigative  Question  Three 

This  investigative  question  addresses  the  “health”  of  the  two  test  equipment 
programs  as  characterized  by  those  involved  in  the  sustainment  of  the  equipment.  This 
protocol  and  series  of  sub-questions  was  developed  to  better  understand  the  major  issues 
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facing  the  equipment  today  (e.g.  obsolescence,  dwindling  operational  capability, 
hardware  shortages,  etc.)  By  asking  the  management  teams  to  identify  the  most  pressing 
issues  facing  their  systems,  one  can  better  understand  any  significant  differences  in  tenns 
of  volume  and  severity  of  future  supportability  challenges.  To  obtain  an  increased  level 
of  objectivity,  this  researcher  also  addressed  these  sub-questions  to  the  lead  Major 
Command  personnel  involved  in  the  field  support  of  this  test  equipment.  Since  these 
individuals  are  in  a  position  to  observe  the  field-level  impact  of  supportability  issues  for 
both  TISS  and  AIS,  they  could,  in  theory,  offer  an  unbiased  perspective  on  the  two 
systems.  One  of  the  most  significant  sub-questions  addresses  the  estimated  timeline  for 
reaching  solutions  to  these  supportability  issues.  Is  there  a  difference  between  the  two 
testers  in  terms  of  a  projected  fix?  What  are  the  obstacles  to  success? 

Investigative  Question  Four 

With  similar  intentions  as  investigative  question  three,  this  protocol  is  aimed  at 
capturing  the  lead  MAJCOM’s  evaluative  perspective  on  the  perfonnance  of  the  two 
management  teams  in  terms  of  field  level  support.  Whereas  the  primary  impetus  behind 
consolidating  test  equipment  for  management  at  a  stand-alone  office  is  one  of  cost 
savings,  this  researcher  contended  that  no  senior  defense  leader  would  wish  to  sacrifice 
support  for  the  warfighter.  Accordingly,  it  is  believed  that  in  order  to  fully  understand 
the  ramifications  of  any  management  strategy,  one  must  certainly  examine  the  impact  of 
that  strategy  on  the  end  users.  In  this  case,  the  end  users  are  the  avionics  technicians  in 
the  field  who  use  this  test  equipment  each  and  every  day.  The  management  staff  at  the 
lead  Major  Command  is  in  a  position  to  observe  and  address  the  frequency  and  rigor  of 
supportability  challenges  faced  by  users  in  the  field.  To  illustrate,  consider  the  impact  of 
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parts  shortages  on  field  technicians. .  .are  those  individuals  more  often  plagued  in  this 
regard  for  one  system  than  another?  Do  they  encounter  unexpected  maintenance 
difficulties  more  often?  In  either  case,  the  aim  of  this  investigative  question’s  protocol  is 
to  examine  the  similarities  and  differences  of  managerial  approach  as  they  affect  the  field 
users  of  the  equipment. 

Data  Collection 

Depending  on  the  specific  qualitative  study  used  for  the  research,  data  can  take 
multiple  forms.  Creswell  (1998)  identifies  the  four  basic  forms  of  qualitative  data  as:  1) 
observations,  2)  interviews,  3)  documents  and  4)  audio-visual  materials  (Creswell, 

1998: 121).  Importantly,  he  notes  that  each  fonn  of  data  has  both  advantages  and 
limitations  that  the  researcher  should  consider  when  planning  the  research  design 
(Creswell,  2003:186-187).  For  this  particular  study,  the  data  collected  consisted 
primarily  of  various  documents,  structured  and  open-ended  interviews.  Examples  of 
documents  reviewed  in  this  study  included  internal  System  Program  Office 
memorandums,  Major  Command  and  depot  equipment  status  briefings  and  integrated 
product  team  meeting  minutes.  Selected  sample  data  obtained  from  interview  notes  and 
reviewed  documents  is  provided  in  Appendices  I  through  P.  Table  3  contains  Creswell ’s 
advantages  and  limitations  for  each  type  of  data  used  in  the  study. 
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Table  3.  Advantages  and  Limitations  for  Data  Collection  Types 


Data  Collection 
Used  in  Study 

Advantages 

Limitations 

Documents 

-  Enables  a  researcher  to  obtain  the 
language  and  words  of  participants 

-  Can  be  accessed  at  a  time  convenient  to 
the  researcher-  an  unobtrusive  source  of 
information 

-  Represents  data  that  are  thoughtful,  in 
that  participants  have  given  attention  to 
compiling 

-  As  written  evidence,  it  saves  a 
researcher  the  time  and  expense  of 
transcribing 

-  May  be  protected  information 
unavailable  to  public  or  private  access 

-  Requires  the  researcher  to  search  out 
the  information  in  hard-to-fmd  places 

-  Requires  transcribing  or  optically 
scanning  for  computer  entry 

-  Materials  may  be  incomplete 

-  The  documents  may  not  be  authentic  or 
accurate 

Interviews 

-  Useful  when  participants  cannot  be 
observed  directly 

-  Participants  can  provide  historical 
information 

-  Allows  researcher  “control”  over  the 
line  of  questioning 

-  Provides  “indirect”  information  filtered 
through  the  views  of  interviewees 

-  Provides  information  in  a  designated 
“place”  rather  than  the  natural  field 
setting 

-  Researcher’s  presence  may  bias 
responses 

-  People  are  not  equally  articulate  and 
perceptive 

(Creswell,  2003:186  to  187) 


In  preparation  for  the  analysis  of  resultant  data,  the  type  and  quantity  of  data 
obtained  for  each  protocol  question  were  carefully  monitored  and  documented.  To 
accomplish  this,  the  researcher  used  the  matrix  shown  in  Appendix  F. 

The  use  of  multiple  data  sources  offers  the  critical  advantage  of  triangulation  when 
seeking  conclusions.  Specifically,  Yin  states  “the  most  important  advantage  presented  by 
using  multiple  sources  of  evidence  is  the  development  of  converging  lines  of 
inquiry. .  .any  finding  or  conclusion  in  a  case  study  is  likely  to  be  much  more  convincing 
and  accurate  if  it  is  based  on  several  different  sources  of  information,  following  a 
corroboratory  mode”  (Yin,  2003:92).  A  diagram  illustrating  the  concept  of  converging 
lines  of  inquiry  is  provided  in  Figure  9. 


64 


Documents 


Archival 

Records 


/ 


FACT 


Open  ended 
interviews 


Observations  Structured  Focused 

(Direct  and  interviews  Interviews 

participant)  and  surveys 


Figure  9.  Convergence  of  Multiple  Sources  of  Evidence  (Yin,  2003:  93) 


Additionally,  Yin  notes  that  the  researcher  may  also  identity  non-convergence,  that 
is,  “multiple  sources  that  nevertheless  address  different  facts”  (Yin,  2003:  92).  The  visual 
depiction  of  non-convergence  is  illustrated  in  Figure  10. 


65 


interviews 

findings 

conclusions 

W 

W 

survey 

findings 

conclusions 

w 

w 

documents 

findings 

conclusions 

w 

w 

Figure  10.  Non-convergence  of  Multiple  Sources  of  Evidence  (Yin,  2003:  93). 


Again,  this  researcher’s  strategy  for  collecting  the  data  was  based  primarily  on 
Creswell’s  (1998)  Data  Collection  Activities  diagram  in  Figure  1 1  (Creswell,  1998:1 10). 
This  diagram  depicts  a  “series  of  interrelated  activities  aimed  at  gathering  good 
information  to  answer  emerging  research  questions”  (Creswell,  1998:110). 

Although  Creswell  points  out  that  a  researcher  may  start  from  any  point  located 
around  the  circle,  he  usually  begins  with  the  “Locating  Site/Individual”  node  (Creswell, 
1998:110). 
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Figure  11.  Data  Collection  Activities  (Creswell,  1998:110) 


1.  Locating  Sites  &  Individuals 

The  motivation  for  this  study  was  to  establish  a  baseline  framework  for  further 
inquiries.  As  a  stepping-off  point,  the  initial  sites  (within  this  study,  defined  as  specific 
management  strategies  for  individual  ATS)  were  limited  only  to  the  existing  USAF 
inventory  of  ATS.  Initial  discussions  and  collaboration  with  advisers  and  subject  matter 
experts  led  to  the  decision  to  examine  the  ATE  within  a  legacy  aircraft  platform  - 
specifically,  F-15  ATE.  In  designing  the  case  study  protocol,  the  relevant  sources  of  data 
were  identified  that  would  be  best  suited  for  a  particular  investigative  question. 

2.  Gaining  Access  &  Making  Rapport 

This  researcher  relied  on  existing  commercial  and  USAF  communication  networks, 
as  well  as  personal  contacts  to  identify  key  management  personnel  within  the  systems 
under  study.  Once  initial  contacts  were  made  within  a  given  equipment  community,  this 


67 


researcher  asked  those  subject  matter  experts  to  identify  additional  personnel  that  might 
yield  further  insight  into  the  research  effort. 

3.  Purposeful  Sampling 

Creswell  (2003)  states  the  objective  of  purposeful  sampling  is  to  “purposefully 
select  participants  or  sites  (or  documents  or  visual  material)  that  will  best  help  the 
researcher  understand  the  problem  and  the  research  question”  (Creswell,  2003:185).  For 
this  study,  we  selected  the  two  subjects  (AIS  and  TISS)  at  the  recommendation  of 
subject-matter  experts  in  the  field.  This  researcher  spoke  with  these  individuals  at  length 
both  in  person  and  over  electronic  communication.  After  providing  them  with  a 
comprehensive  background  on  the  aim  of  the  study,  both  individuals  concurred  on  the 
two  selected  systems  as  suitable  candidates  for  this  research. 

4.  Collecting  Data 

As  previously  mentioned,  data  were  collected  from  documents  and  interviews.  The 
documents  primarily  came  from  managers  at  the  System  Program  Office,  depot  and 
Major  Command  levels.  Prior  to  conducting  interviews,  this  researcher  requested  and 
obtained  approval  for  the  use  of  volunteers  in  research  from  the  Air  Force  Research 
Laboratory  Human  Use  Administrator. 

5.  Recording  Information 

When  possible,  during  targeted  interviews,  we  employed  a  third-party  observer  to 
transcribe  data  during  interviews.  This  practice  provided  additional  rigor  to  the  accuracy 
of  data  collected.  Furthermore,  interviewees  were  provided  with  summary  transcripts  of 
the  collected  data  to  verify  accuracy  and  authenticity. 
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6.  Resolving  Field  Issues 

This  researcher  maintained  open  communication  with  field  personnel  and 
interviewees  throughout  the  research  effort.  Existing  or  unresolved  field  issues  are 
discussed  in  chapter  five  of  this  report. 

7.  Storing  Data 

Multiple  copies  of  the  data  collection  were  made  in  order  to  prevent  the  accidental 
deletion  or  corruption  of  files.  When  warranted,  hard  copies  of  the  infonnation  were 
printed  and  stored  with  the  rest  of  the  research  documentation. 

Validity  and  Reliability 

The  last  area  of  research  design  to  be  addressed  is  the  validity  and  reliability  of  the 
research  method.  Validity  is  described  by  Leedy  et  al  (2001)  as  “the  accuracy, 
meaningfulness,  and  credibility  of  the  research  project  as  a  whole”  (Leedy  et  al, 

2001 : 103).  Reliability,  however,  is  defined  as  “the  consistency  with  which  a  measuring 
instrument  yields  a  certain  result  when  the  entity  being  measured  hasn’t  changed”  (Leedy 
et  al,  2001:31). 

In  dealing  with  validity  and  reliability,  four  tests  have  been  identified  as  being 
relevant  to  case  study  research:  construct  validity,  internal  validity,  external  validity,  and 
reliability  (Yin,  2003:33).  Table  4  outlines  the  four  tests  and  tactics  used  to  address  each 
of  the  tests.  The  validity  and  reliability  of  this  study  will  be  discussed  at  length  within 
Chapter  four. 
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Table  4.  Case  Study  Tactics  for  Four  Design  Tests 


Tests 

Case  Study  Tactic 

Phase  of  research  in 
which  tactic  occurs 

Construct  Validity 

-  Use  multiple  sources  of  evidence 

-  Establish  chain  of  evidence 

-  Have  key  informants  review  draft  of  case 
study  report 

Data  collection 

Data  collection 
Composition 

Internal  Validity 

-  Do  pattern-matching 

-  Do  explanation-building 

-  Address  rival  explanations 

-  Use  logic  models 

Data  analysis 

Data  analysis 

Data  analysis 

Data  analysis 

External  Validity 

-  Use  theory  in  single-case  studies 

-  Use  replication  logic  in  multiple-case 
studies 

Research  design 

Research  design 

Reliability 

-  Use  case  study  protocol 

-  Develop  case  study  database 

Data  collection 

Data  collection 

(Yin,  2003:34) 


Summary 

This  chapter  described  the  research  design  and  specific  methodology  selected  to 
conduct  the  thesis  study.  A  multiple-case  study  design  was  chosen  for  researching  the 
management  strategies  in  practice  for  the  oversight  of  TISS  and  AIS.  The  data  were 
collected  primarily  from  archival  data,  documents  and  interviews.  Once  all  the  relevant 
information  was  identified,  this  researcher  used  a  variety  of  case  study  analytical  tools, 
including  pattern-establishment,  categorical  aggregation  and  direct  interpretation  for  the 
purposes  of  trend  identification  and  theory  building.  Next,  Chapter  4  will  discuss  the 
findings  of  the  research. 
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IV.  Analysis  and  Results 


Chapter  Overview 

Within  this  chapter,  we  present  and  explain  the  analysis  and  results  of  this  case 
study.  First,  the  chapter  discusses  the  answers  to  the  investigative  questions  posed  at  the 
beginning  of  the  study,  each  of  which  are  followed  by  a  summary  of  specific  findings. 
Prior  to  the  chapter’s  conclusion,  issues  of  validity  and  reliability  are  addressed  for  this 
study. 

Investigative  Question  One 

What  are  the  differences  in  how  Automated  Test  Equipment  is  being  managed? 

In  preparing  for  this  study,  it  was  theorized  that  there  may  have  been  dissimilarity 
in  the  day-to-day  management  techniques  and  procedures  performed  by  members  of  the 
System  Program  Office  (TISS)  and  depot  (AIS)  staffs.  Appendix  A  outlines  the  specific 
case  study  protocol  questions  that  are  relevant  to  this  investigative  question. 
Management  Structure 

Both  programs  utilize  an  integrated  product  team  (IPT)  management  structure  to 
oversee  the  management  of  their  equipment.  The  integrated  product  team  structure 
involves  different  team  members  as  specialists  in  a  given  area  to  collectively  address 
managerial  issues  of  concern.  Examples  of  positions  represented  on  an  integrated 
product  team  include:  program  manager,  logistics  manager,  item  manager,  equipment 
specialist,  engineer,  contracting  officer,  equipment  contractors,  Major  Command  staff 
members  and  technical  training  school  representatives.  (See  Appendices  I,  J) 
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Collectively,  these  individuals  determine  and  provide  solutions  to  field  users  of  their 
respective  equipment  as  they  arise.  Varying  groups  of  these  individuals  may  meet 
together  at  different  intervals  to  discuss  new  developments  and  managerial  actions.  All 
respondents  from  the  System  Program  Office  indicated  that  the  field  user’s  perspective 
was  genuinely  sought  and  considered  when  managerial  actions  were  being  taken 
regarding  engineering  or  logistical  solutions.  (See  Appendix  I)  One  significant 
difference  observed  during  the  collection  of  data  was  the  “involvement  by  proximity”  of 
a  lead  Major  Command  liaison  at  the  System  Program  Office.  This  individual  is  assigned 
to  the  F-15  acquisition  System  Program  Office  as  a  direct  interface  for  the  lead  Major 
Command  F-15  avionics  staff,  and  is  largely  responsible  for  monitoring  progress  on 
Major  Command  interests  currently  in  development.  Although  the  TISS  program  is  in 
sustainment  phase,  its  primary  management  is  collocated  with  the  acquisition  System 
Program  Office,  and  the  lead  Major  Command  liaison  is  physically  present  to  serve  as  an 
advocate  for  TISS  sustainment  issues  and  decisions  that  affect  field  units.  (See  Appendix 
I)  There  is  not  an  equivalent  position  at  the  depot,  as  the  Major  Command  liaison  is 
primarily  targeted  towards  monitoring  programs  in  the  acquisition,  not  sustainment, 
phase  of  product  development. 

Planning,  Obsolescence  and  the  Impact  of  Next  Generation  Equipment 

The  presence  of  a  near-tenn  replacement  for  certain  components  of  the  AIS  has  an 
impact  on  the  intensity  and  level  of  planning  activities  performed  by  its  managers. 
Specifically,  there  is  another  test  equipment  system,  known  as  the  Electronic  Systems 
Test  Set  (ESTS),  which  has  been  designed  and  introduced  as  a  replacement  system  for  six 
of  the  eight  components  of  the  Avionics  Intennediate  Shop.  (See  Appendix  J)  The 
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Electronic  Systems  Test  Set  began  development  in  1992  and  was  first  introduced  to  the 
field  in  2000,  but  encountered  obsolescence  and  diminishing  manufacturing  source  issues 
before  it  was  fielded.  This  researcher  learned  from  avionics  personnel  that  while  the 
Electronic  Systems  Test  Set  is  being  used  in  the  field,  it  has  not  yet  fully  replaced  in 
practice  the  six  intended  components  of  AIS  -  indeed,  these  are  also  being  used  by  field 
avionics  technicians  to  meet  mission  demands.  (See  Appendix  J)  Unfortunately,  despite 
the  shortcomings  of  the  Electronic  Systems  Test  Set,  its  existence  as  the  intended  next- 
generation  replacement  has  had  a  paralyzing  effect  on  funding  support  for  the  original 
AIS  testers.  Not  surprisingly,  it  would  seem  that  Air  Force  decision-makers  are 
disinclined  to  offer  continued  support  for  a  system  that,  in  principal,  has  already  been 
“replaced”.  The  specific  impact  and  contribution  of  the  Electronic  Systems  Test  Set  to 
the  F-15  avionics  community  is  beyond  the  scope  of  this  research,  but  additional 
implications  of  the  Electronic  Systems  Test  Set,  its  commodity  consignment  and 
obsolescence  are  discussed  as  a  possibility  for  future  research  in  chapter  five. 

A  more  pressing  concern  for  the  AIS  management  team  is  the  sustainment  of  the 
other  two  components,  the  Antenna  Test  Station  and  Enhanced  Aircraft  Radar  Test 
Station,  both  of  which  are  not  being  replaced  by  the  Electronic  Systems  Test  Set  and 
must  be  sustained  until  2025.  (See  Appendices  J,  L)  AIS  managers  and  lead  Major 
Command  avionics  personnel  have  already  posited  several  scenarios  for  mission 
degradation  as  a  result  of  projected  obsolescence;  it  is  expected  that  a  continuation  of 
nonnal  obsolescence  rates  will  bear  witness  to  aircraft  groundings  by  2007  (Elliott, 

2004).  Respondents  cited  a  variety  of  reasons  for  this  projected  impact,  including  limited 
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repair  sources,  inability  to  procure  component  level  parts  and  limited  inventory  spares. 
(See  Appendices  J,  N) 

By  contrast,  the  TISS  program  is  in  the  midst  of  a  sizeable  technology  insertion 
program  to  upgrade  the  existing  inventory  of  35  test  stations.  At  the  time  of  this 
publication,  the  successful  execution  of  this  technology  insertion  program  is  the  primary 
focus  of  the  TISS  integrated  product  team.  Just  as  the  Antenna  Test  Station  and 
Enhanced  Aircraft  Radar  Test  Station  management  team  is  doing  currently,  the  TISS 
management  team  perfonned  internal  case  study  projections  in  2002  that  projected  a  loss 
in  mission  capability  in  2007.  After  completing  these  studies,  the  TISS  program  failed  to 
obtain  funding  approval  in  the  2002  program  objective  memorandum,  but  secured  the 
requested  funding  two  years  later.  Additionally,  the  TISS  Technology  Insertion  Program 
input  was  one  of  only  two  F-15  inputs  to  receive  funding  approval  in  the  2004  program 
objective  memorandum.  (See  Appendix  I) 

The  main  objectives  of  the  TISS  Technology  Insertion  Program  were  to  replace 
commercial  off-the-shelf  (COTS)  equipment  from  the  original  TISS  stations  dating  from 
1988  to  1992,  as  well  as  transition  to  VXI  (Versa  Module  Eurocard  Extensions  for 
Industry)  card-based  technology.  First,  a  wide  array  of  commercial-off-the-shelf 
component  manufacturers  advised  TISS  managers  that  their  components  would  not  be 
supported  beyond  2007.  According  to  collected  interview  data,  the  TISS  Technology 
Insertion  Program  addresses  and  solves  this  problem.  (See  Appendix  I)  Secondly,  put  in 
simple  terms,  card-based  technology  capitalizes  on  the  use  of  smaller  “plug  and  play” 
card  components  instead  of  entire  drawers  in  rack  test  equipment  systems.  At  the 
completion  of  the  TISS  Technology  Insertion  Program,  the  number  of  bays  in  a  complete 
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TISS  station  will  be  reduced  from  six  to  three.  This  will  provide  logistical  benefits  in 
terms  of  shipping  and  transportation  repair  (less  cost  to  ship  a  card  versus  a  drawer)  while 
also  decreasing  the  overall  size  of  the  station  (deployable  footprint).  More  specific 
discussion  of  TISS  Technology  Insertion  Program  funding  and  execution,  as  well  as  F-15 
test  equipment  funding  matters  in  general  is  addressed  below  in  the  findings  for 
Investigative  Question  Two. 

To  summarize,  the  impact  of  obsolescence  appeared  to  be  a  much  greater  and  more 
persistent  concern  for  the  AIS  management  team  than  the  TISS  management  team.  (See 
Appendices  I,  J,  K,  L,  M,  N)  The  disparity  in  system  age  is  likely  a  major  contributor  to 
this  fact  (AIS  was  first  introduced  in  the  early  1970s,  while  TISS  was  initially  fielded  in 
1992),  but  there  was  also  a  perceived  difference  in  the  management  teams’  assessment  of 
their  own  greatest  challenges.  The  AIS  management  team  overwhelmingly  cited 
obsolescence  as  their  greatest  challenge,  whereas  the  TISS  respondents  instead  suggested 
that  the  overall  complexity  of  the  supported  electronic  warfare  components  was  the 
greatest  challenge.  (See  Appendices  I,  J)  Both  programs,  however,  stressed  the 
importance  of  forecasting  and  planning  as  the  best  means  for  combating  parts  and 
equipment  obsolescence.  Additionally,  both  described  the  development  of  studies 
projecting  shortfalls  in  operational  capability  or  aircraft  groundings  as  essential  to  being 
competitive  in  the  funding  approval  process.  More  specific  commentary  on  the  topics  of 
management  planning  and  obsolescence  will  be  provided  in  chapter  five. 

Communication  and  Information  Flow 

Two  significant  differences  were  observed  between  the  two  test  equipment 
programs  that  relate  to  communication.  First,  the  TISS  community  maintains  and  uses 
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extensively  a  secure  website  for  the  documentation  of  historical  and  ongoing 
management  issues.  This  website  is  broad  in  scope,  containing  information  regarding 
field  problem  reports,  software  configuration,  personnel  contacts,  meeting  minutes  and 
system  history.  The  website  is  secure  and  accessible  only  by  user  login  and  password. 

All  personnel  who  are  part  of  the  TISS  integrated  product  team  have  access  to  this 
resource.  (See  Appendix  I)  This  researcher  observed  the  maintenance  and  upkeep  of  this 
website  as  not  only  a  valuable  communication  tool,  but  also  an  excellent  historical 
reference. 

Additionally,  the  TISS  program  is  eligible  to  utilize  the  Air  Warfare  Center  Bulletin 
Board  System  (AWC  BBS)  at  Eglin  AFB,  Florida  for  the  distribution  of  all  Test  Program 
Set  (TPS)  software  upgrades.  This  is  a  classified  system  used  by  both  the  Air  Warfare 
Center  and  the  Electronic  Warfare  Division  of  the  Wamer-Robins  Air  Logistics  Center  to 
send  out  classified  aircraft  flight  programs.  As  a  result,  all  F-15  field  units  have  access. 
Essentially,  this  system  allows  users  in  the  field  to  rapidly  download  entire  updates  to  test 
program  sets.  Typically,  field  units  wait  days,  weeks  or  months  for  hard  copies  of 
software  to  arrive  via  traditional  transportation  methods  before  manually  loading  the 
updates  into  their  systems.  The  use  of  the  Air  Warfare  Center  Bulletin  Board  System 
greatly  accelerates  this  process.  The  TISS  program  is  able  to  take  advantage  of  this 
mechanism  because  the  majority  of  TISS  software  is  classified.  (See  Appendix  I)  The 
same  cannot  be  said  for  the  Avionics  Intennediate  Shop  software;  as  a  result,  its 
managers  are  not  privy  to  its  use. 
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Synopsis  of  Investigative  Question  One 

What  are  the  differences  in  how  ATE  is  being  managed? 

The  TISS  program  benefits  from  the  presence  of  a  lead  Major  Command  liaison 
working  in  the  same  facility  as  the  primary  management  personnel 
Planning  efforts  are  impacted  by  the  two  programs’  different  positions  along  the 
product  life-cycle  timeline.  TISS  has  secured  funding  to  combat  projected  shortfalls 
in  operational  capability;  the  AIS  has  suffered  in  this  regard  due  to  the  apparent 
stagnation  of  the  Electronic  Systems  Test  Set  program 

Obsolescence  issues,  while  significant  for  both  programs,  are  a  more  urgent  concern 
for  the  AIS  program,  particularly  for  the  Antenna  Test  Station  and  Enhanced  Aircraft 
Radar  Test  Station  -  components  which  must  be  maintained  until  2025. 

The  TISS  program  benefits  from  having  an  extensive  internal  website,  as  well  as  the 
use  of  the  Air  Warfare  Center  Bulletin  Board  System  secure  server  for  test  program 
software  downloads  to  field  users  -  AIS  does  not  have  similar  systems. 

Investigative  Question  Two 

How  much  sustainment  funding  is  budgeted/funded  in  the  Program  Objective 
Memorandum  (POM)  and  Future  Year  Defense  Plan  (FYDP)? 

The  objective  of  this  investigative  question  was  largely  the  same  as  that  of 
investigative  question  one,  only  in  terms  of  funding.  Primarily,  the  aim  was  to  identify 
whether  a  significant  difference  existed  between  the  two  programs  as  it  pertains  to 
requested  and  received  sustainment  funds.  Appendix  B  outlines  the  specific  case  study 
protocol  questions  that  are  relevant  to  this  investigative  question. 
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" Colors  ”  of  Money 

During  the  course  of  this  study,  several  distinct  categories  of  funding  were 
identified  that  are  commonly  used  within  the  test  equipment  community.  These 
categories  are  identified  in  Table  5. 


Table  5.  Categories  of  Funding  Relevant  to  TISS  and  AIS 


EEIC 

Description 

583 

DPEM  Sustaining  Engineering  money  used  to  investigate  system  level  equipment  problems 

MSD  Eng 

Used  to  investigate  component  level  problems  that  aren't  considered  "system"  level 

MSD  UUT 

Example:  a  multimeter  in  a  multi-bay  rack  system  would  use  MSD  funds;  the  rack  would  use  583  $$ 

Cl 

CD 

540 

Depot  Programmed  Equipment  Maintenance  (DPEM)  money  used  for  software  maintenance 

n 

545 

DPEM  money  used  to  maintain  exchangeables  (core  chargeable  equipment) 

Repairs 

Material  Support  Division  Repair  -  from  the  Air  Force  Working  Capital  Stock  Fund 

Spares 

Material  Support  Division  Buy  (Spares)  -  from  the  Air  Force  Working  Capital  Stock  Fund 

CL 

3600 

Development  Funds 

w 

3010 

Production  Funds 

For  both  of  the  test  equipment  programs  under  consideration  in  this  study,  Air 
Force  Materiel  Command  is  responsible  for  Material  Support  Division  Engineering,  Unit- 
under-test,  540  and  545  money,  while  the  lead  Major  Command  (ACC),  Air  National 
Guard  and  Air  Force  Reserve  Command  are  responsible  for  funding  583  requirements. 
The  3600  and  3010  development  and  production  monies  flow  directly  through  individual 
program  offices  -  in  this  case,  the  F-15  System  Program  Office. 

Some  difficulty  in  obtaining  specific  funding  data  for  both  programs  was 
encountered  during  this  study.  This  difficulty  is  addressed  in  greater  detail  within  chapter 
five.  Nevertheless,  there  are  several  significant  differences  in  terms  of  funded  and 
unfunded  requirements  for  the  two  programs.  First,  the  TISS  Technology  Insertion 
Program  is  wholly  funded  through  2007,  with  managers  working  on  2008  program 
objective  memorandum  inputs  at  the  time  of  this  study’s  publication.  The  TISS 
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Technology  Insertion  Program  received  $41.6M  from  the  2004  Program  Objective 
Memorandum,  of  which  approximately  $36M  is  targeted  for  the  production  and 
development  contract  with  the  manufacturer,  with  the  remaining  funds  dedicated  to 
engineering  support  and  various  program  office  operations.  The  $36M  will  be,  for 
practical  purposes,  evenly  distributed  between  3010  production  funds  and  3600 
development  funds.  Incidentally,  it  should  be  noted  that  the  Technology  Insertion 
Program  was  placed  on  the  lead  Major  Command’s  mission  essential  funding  list  for  the 
2004  program  objective  memorandum,  and  would  become  one  of  only  two  F-15  program 
inputs  to  receive  approval.  (See  Appendix  K) 

By  contrast,  the  AIS  program  has  a  wide  array  of  unfunded  requirements.  AIS 
respondents  related  the  following  inadequacies  in  terms  of  recent  depot  funding  trends: 
for  Material  Support  Division  Engineering,  funding  has  been  at  approximately  30%  of 
the  actual  requirement;  for  540  money,  funding  has  been  at  approximately  80%  of  the 
actual  requirement;  and  for  583  money,  funding  has  been  at  approximately  15%  of  the 
actual  requirement.  This  is  in  stark  contrast  to  the  equivalent  figures  for  TISS  depot 
sustainment  funding,  where  respondents  reported  receiving  all  requested  monies,  with 
one  exception  during  fiscal  year  2005.  (See  Appendix  L) 

To  draw  more  specific  comparisons,  the  specific  depot  funding  amounts  for  fiscal 
years  2001  through  2005  were  obtained,  compiled  and  analyzed  for  both  programs.  A 
summary  of  these  values  and  analysis  is  included  in  Appendix  E.  This  analysis  identified 
several  points  that  may  suggest  a  historical  funding  difference  between  the  TISS  and  AIS 
programs. 
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First,  a  simple  trend  examination  of  the  data  suggests  that  TISS  sustainment  funding 
at  the  depot  has  been  more  steady  and  consistent  than  that  of  AIS.  Figures  12  and  13 
illustrate  this  difference.  For  TISS,  the  dollar  values  in  total  suggest  a  continued  upward 
trend  in  total  funding,  with  individual  categorical  funding  remaining  largely  consistent 
through  the  data  range. 


Historical  TISS  Funding 


— ♦ —  583 
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Figure  12.  Historical  TISS  Sustainment  Funding,  FY01  -  FY05 


By  contrast,  the  trend  for  AIS  funding  has  been  much  more  erratic,  indicated  in 
Figure  13.  Individual  categorical  funding  decreased,  only  to  increase  again,  with 
tremendous  funding  spikes  emerging  in  fiscal  year  2005. 
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Historical  AIS  Funding 
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Figure  13.  Historical  AIS  Funding,  FY01  -FY05. 


It  is  interesting  to  note  that  the  largest  categories  of  TISS  funding  were  focused  on 
the  procurement  of  spares  and  system  repairs,  while  the  bulk  of  AIS  funding  was  targeted 
towards  sustaining  engineering  funds.  This  would  suggest  that  the  AIS  program  has  a 
greater  need  than  TISS  to  analyze  component  problems  and  develop  solutions  for 
obsolescence.  (See  Appendices  K,  L)  Most  importantly,  however,  is  the  success  the 
TISS  program  has  had  in  terms  of  accessing  and  obtaining  3010  and  3600  dollars  from 
the  F-15  program  office.  In  effect,  Program  Objective  Memorandum  approval  for  the  in¬ 
work  technology  insertion  program  has  enabled  TISS  managers  to  preempt  potential 
obsolescence  problems  in  the  coming  years. 

Managerial  Perspectives  on  Adequate  Funding 

Perhaps  the  most  dramatic  difference  in  regards  to  program  funding  is  found  within 
respondents’  own  perspectives  on  the  adequacy  or  inadequacy  of  funding  support. 
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Simply  put,  respondents  from  the  TISS  program  feel  that  they  are  receiving  adequate 
funding,  while  respondents  from  the  AIS  program  do  not.  In  three  interviews,  the  AIS 
program  managers  were  most  concerned  with  inadequate  funding  to  support  the  Antenna 
Test  Station  and  Enhanced  Aircraft  Radar  Test  Station  -  the  two  members  of  the 
Avionics  Intermediate  Shop  that  are  not  being  replaced  by  the  Electronic  Systems  Test 
Set  and  must  be  sustained  until  2025.  It  was  noted  that  fiscal  year  2005  depot 
sustainment  dollars  showed  a  sharp  increase  for  the  AIS  program  over  the  previous  year’s 
funding  -  specifically,  $22  million. .  .an  increase  of  more  than  300%.  (See  Appendices  K, 
L)  While  further  research  will  be  required  to  detennine  if  this  increase  is  an  outlying 
occurrence  or  the  start  of  a  new,  upward  trend,  it  is  quite  possible  that  the  dramatic 
change  is  the  result  of  increased  testimonials  and  case  studies  on  behalf  of  the  AIS, 
Antenna  Test  Station,  and  Enhanced  Aircraft  Radar  Test  Station  community. 

Synopsis  of  Investigative  Question  Two 

How  much  sustainment  funding  is  budgeted/funded  in  the  Program  Objective 
Memorandum  (POM)  and  Future  Year  Defense  Plan  (FYDP)  ? 

Historical  depot  sustainment  funding  data  was  obtained  for  fiscal  years  2001  through 
2005;  these  values  are  included  in  Appendix  E. 

The  TISS  program  received  approval  in  the  2004  Program  Objective  Memorandum 
for  a  $41.6M  technology  insertion  program  to  replace  obsolescent  commercial-off- 
the-shelf  equipment  and  reduce  the  logistical  footprint  of  all  existing  test  stations 
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The  AIS  program  has  not  yet  secured  funding  to  address  growing  obsolescence 
concerns  for  the  sustainment  of  the  Antenna  Test  Station  and  Enhanced  Aircraft 
Radar  Test  Station  through  2025 

Overall  sustainment  funding  appears  to  have  been  more  consistent  for  TISS  than  for 
AIS 

Investigative  Question  Three 

What  are  System  Program  Office,  depot  and  Major  Command  assessments  of  Automatic 
Test  Equipment  sustainability  for  the  two  programs? 

This  investigative  question  was  intended  to  explore  the  unique  circumstances 
facing  each  test  equipment  program  as  they  pertain  to  near-  and  long-term  sustainability. 
This  researcher  collected  interview  data,  program  documents  and  archival  figures  from 
program  managers  and  lead  Major  Command  personnel  to  assess  the  nature  and  severity 
of  sustainability  issues  facing  the  program,  as  well  as  managers’  plans  for  addressing 
these  problems.  Additionally,  this  researcher  sought  to  compare  estimated  timelines  for 
reaching  a  solution.  Appendix  C  outlines  the  specific  case  study  protocol  questions  that 
are  relevant  to  this  investigative  question. 

Operational  Capability  and  Diminishing  Manufacturing  Sources 

Both  test  equipment  program  managers  and  Major  Command  personnel  cited  the 
impact  of  obsolescence  and  hardware  shortages  as  the  most  significant  sustainability 
issues.  A  significant  difference,  however,  was  observed  in  the  management  teams’ 
overall  outlook  on  the  prognosis  for  addressing  these  issues.  First,  in  regards  to  the  TISS 
program,  the  only  specific  hardware  or  component  shortage  issues  are  not  terribly  severe, 
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and  are  inclusively  addressed  as  part  of  the  approved  technology  insertion  program, 
which  is  set  for  implementation  in  2007.  TISS  respondents  cited  one  power  supply 
sustainment  issue  in  particular,  but  noted  that  funding  had  already  been  secured  to  obtain 
suitable  spares  that  will  sufficiently  cover  user  demand  until  the  technology  insertion 
program  is  in  place.  (See  Appendix  M) 

On  the  contrary,  the  component  sustainability  issues  facing  the  AIS  program  are 
significantly  impacting  tester  operational  capability.  Specifically,  current  operational 
levels  for  the  AIS  are  approaching  fifty  percent.  Interview  respondents  indicated  that 
thirty  percent  of  major  subassemblies  for  the  Avionics  Intennediate  Shop  can  no  longer 
be  procured,  with  component  repair  turnaround  times  in  excess  of  120  days.  As  a  result, 
the  throughput  of  units  under  test  (i.e.  components  from  the  aircraft)  is  slowed 
considerably.  These  issues  are  the  focus  of  ongoing  efforts  on  the  part  of  AIS  staff 
members  and  Major  Command  personnel  to  secure  funding.  (See  Appendix  N) 

In  regards  to  the  prolonged  severity  of  these  sustainability  issues,  there  is  again  a 
dramatic  difference  between  the  two  test  equipment  programs.  Several  TISS  respondents 
indicated  that  they  are  only  recently  beginning  to  encounter  obsolescence  issues,  while 
AIS  respondents  indicated  that  these  issues  have  been  ongoing  for  approximately  fifteen 
years.  (See  Appendices  M,  N)  Additionally,  a  review  of  each  program’s  history  suggests 
another  difference  in  terms  of  tester  modifications  and  upgrades. 

TISS  was  initially  fielded  in  1992  and,  in  June  of  2002,  received  funding  in  the 
2004  program  objective  memorandum  for  an  extensive  upgrade  through  a  technology 
insertion  program  -  an  elapsed  period  of  ten  years.  (See  Appendix  I)  By  comparison,  the 
Antenna  Radar  Test  Set,  one  of  the  components  of  AIS  slated  for  sustaimnent  until  2025, 
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was  fielded  in  the  early  1970s  and  has  been  modified  only  once,  in  1985.  (See  Appendix 
N)  Nearly  twenty  years  have  elapsed  without  any  significant  improvements. 

Respondents  from  both  the  lead  Major  Command  and  the  AIS  program  both  responded 
that  inadequate  funding  was  the  primary  roadblock  to  the  timely  resolution  of 
sustainability  issues.  (See  Appendices  N,  O) 

Synopsis  of  Investigative  Question  Three 

What  are  System  Program  Office,  depot  and  Major  Command  assessments  of 
Automatic  Test  Equipment  sustainability  for  the  two  programs? 

Obsolescence  and  diminishing  manufacturing  sources  are  the  primary  issues  plaguing 
the  two  test  equipment  programs 

The  AIS  program  faces  a  greater  severity  of  obsolescence  challenges 

Funding  is  generally  considered  to  be  the  primary  hurdle  to  overcome  in  the  timely 

resolution  of  obsolescence  and  sustainment  issues  for  both  testers 

Investigative  Question  Four 

What  are  Major  Command  assessments  of  field  units  ’  abilities  to  support  assigned 
equipment  with  resources  provided  by  the  System  Program  Office  and  depot? 

This  intent  of  this  investigative  question  was  to  capture  the  impact  of  test 
equipment  management  teams  on  the  perfonnance  of  field  units  and  avionics 
maintenance  personnel.  It  was  theorized  that  lead  Major  Command  representatives 
would  have  a  broad  perspective  and  open  access  to  the  successes  and  difficulties 
encountered  by  field  units,  and  would  serve  as  an  objective  judge  of  mission  impact.  For 
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this  investigative  question,  this  researcher  relied  almost  exclusively  on  interview  data 
collected  from  lead  Major  Command  personnel.  Appendix  D  outlines  the  specific  case 
study  protocol  questions  that  are  relevant  to  this  investigative  question. 

Candid  Feedback  and  Objectivity 

First  and  foremost,  it  should  be  noted  that  a  degree  of  resistance  was  encountered 
from  respondents  in  regards  to  the  protocol  questions  associated  with  this  investigative 
question.  This  researcher  expects  that  this  is  due  to  the  candid  nature  of  these  particular 
protocol  questions,  and  perhaps  the  respondents  were  uncomfortable  with  voicing 
particular  opinions.  Additional  commentary  on  this  experience  is  discussed  further 
within  chapter  five. 

Favorable  Characterization  of  Field  Unit  Equipment  Support 

All  in  all,  Major  Command  respondents  commented  favorably  on  the  ability  of 
field  units  to  support  their  assigned  test  equipment,  be  it  TISS  or  AIS  components.  (See 
Appendix  P)  When  constructing  the  protocol  for  this  portion  of  the  study,  there  was  a 
particular  interest  in  determining  the  frequency  with  which  field  technicians  are  forced  to 
rely  on  creative,  higher-authority  (e.g.  System  Program  Office  or  depot)  directed 
troubleshooting  or  maintenance  procedures  due  to  shortcomings  in  existing  technical 
data.  The  reason  for  this  interest  was  due  to  this  researcher’s  previous  personal 
experience  within  an  Air  Force  maintenance  unit,  where  the  observation  of  such 
occurrences  was  not  uncommon.  Nevertheless,  the  study’s  respondents  indicated  that  the 
propensity  for  such  an  occurrence  within  the  F-15  avionics  community  is  small. 
Furthermore,  respondents  indicated  that  when  such  a  situation  does  occur,  Major 
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Command  personnel  liaise  with  both  SPO  and  depot  managers  to  determine  the  best 
course  of  action. 

Technical  orders  and  time  compliance  technical  orders  were  characterized  to  be  of 
good  quality  for  both  programs,  but  respondents  stressed  the  importance  of  coordination 
between  all  involved  parties  for  success.  Additionally,  respondents  described  the  level  of 
troubleshooting  support  for  both  programs  as  adequate.  (See  Appendix  P)  Again,  it  is 
important  to  note  that  a  lack  of  elaboration  was  observed  as  far  as  specific  or  judgmental 
comments  on  the  support  provided  by  either  program,  primarily  from  one  respondent. 

The  final  aspect  of  this  investigative  question  revolved  around  the  urgency  and 
frequency  of  supplied  parts  and  components  to  field  units.  It  was  theorized  that  a  strong 
disparity  between  the  two  programs  (i.e.  much  longer  backorder  periods  for  one  system 
versus  the  other)  would  suggest  a  difference  in  terms  of  top-level  management  support. 
Data  collected,  however,  and  comments  received  from  respondents  did  not  reflect  any 
dissimilarity  in  this  regard  between  the  two  programs. 

Perceptions  and  Additional  Thoughts 

Having  addressed  the  issues  of  apparent  reluctance  and  candor,  a  brief  discussion 
of  additional  thoughts  on  this  investigation  are  now  presented.  While  specific  supporting 
data  were  not  obtained  in  the  relevant  data  collection,  there  are  a  number  of  research 
observations  made  through  other  portions  of  the  study  that  seem  to  suggest  a  distinction 
between  the  supportability  of  the  two  programs. 

First,  as  previously  mentioned,  several  TISS  respondents  commented  on  the 
importance  of  their  internally-managed  website  as  a  communication  tool  for  distributing 
information  amongst  both  management  personnel  and  users  in  the  field.  This  website,  in 
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serving  as  a  forum  for  the  exchange  of  ideas  and  posting  of  timely  and  pertinent 
information,  would  seem  to  have  direct  relevance  to  the  TISS  management’s  ability  to 
support  users  in  the  field,  but  no  respondents  from  the  lead  Major  Command  made 
mention  of  this  tool.  A  separate  discussion  with  one  of  the  TISS  managers  provides  the 
following  scenario.  A  field  user  encounters  a  supportability  problem  or  issue  and 
contacts  a  manager  at  the  System  Program  Office.  The  manager  then  works  in 
coordination  with  other  members  of  the  integrated  product  team  to  resolve  this  issue. 
During  this  period,  a  synopsis  of  the  field  issue,  as  well  as  relevant  instructions  for  field 
users  on  interim  workarounds  are  posted  on  the  community  website.  This  scenario 
arguably  demonstrates  a  high  level  of  support  for  technicians  in  the  field  and  is  certainly 
applicable  to  the  objectives  set  forth  within  the  fourth  investigative  question.  (See 
Appendix  I) 

Additionally,  the  apparent  disparity  between  TISS  and  AIS  in  tenns  of  parts 
availability  and  lengthy  repair  turnaround  was  expected  to  be  an  indicator  of  varying 
levels  of  support  for  technicians  in  the  field.  TISS  respondents  indicated  only  minor 
obsolescence  challenges,  while  AIS  managers  revealed  very  high  non-availability  rates 
for  subassemblies,  as  well  as  excessive  repair  turnaround  times.  (See  Appendix  J  and  L) 
The  specific  reasons  aside,  user  supportability  should  certainly  be  linked  to  this  disparity, 
as  the  integrated  product  teams  comprise  the  very  personnel  charged  with  pursuing 
solutions  to  these  issues.  Still,  lead  Major  Command  respondents  did  not  suggest  this 
disparity  as  a  contributing  factor  to  one  test  equipment  program  being  more  supportable 
than  another,  instead  indicating  that  field  users  of  both  testers  get  the  parts  they  need  in  a 
timely  fashion.  (See  Appendix  P) 


All  told,  after  reviewing  collected  data  for  the  first  three  investigative  questions,  it 
was  expected  that  a  supportability  disparity  would  be  detected  regarding  Investigative 
Question  Four.  Specifically,  it  was  expected  that  TISS  users  would  enjoy  a  higher  level 
of  supportability  than  users  of  the  AIS.  Collected  data  from  the  lead  Major  Command, 
however,  does  not  conclusively  support  this  expectation,  despite  an  array  of  related 
disparities  found  among  data  generated  from  the  other  investigative  questions.  Future 
research  should  be  performed  to  investigate  this  phenomenon  in  greater  depth. 

Synopsis  of  Investigative  Question  Four 

What  are  Major  Command  assessments  of  field  units  ’  abilities  to  support 
assigned  equipment  with  resources  provided  by  the  System  Program  Office  and  depot? 
The  overall  characterization  of  field  units’  abilities  to  maintain  TISS  and  AIS  was 
positive 

This  researcher  detected  a  possible  reluctance  on  the  part  of  respondents  to  offer 
judgmental  or  critical  assessments  on  the  programs  under  study 
Critical  analysis  of  data  collected  for  the  other  investigative  questions  seems  to 
suggest  a  supportability  disparity  between  the  two  systems  that  was  not  traceable  to 
collected  Major  Command  responses 

Research  Findings 

The  collected  data  suggest  that  there  are  significant  differences  between  the  two  test 
equipment  programs,  particularly  within  the  areas  targeted  by  investigative  questions  one, 
two,  and  three.  The  findings  from  investigative  question  four  are  inconclusive.  Specific 
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differences  within  each  protocol  question  are  identified  within  the  matrix  located  in 
Appendix  G. 

Validity  and  Reliability 

As  introduced  in  chapter  three,  it  is  necessary  to  discuss  the  validity  and  reliability 
of  these  results.  Again,  Yin’s  three  categories  of  validity  are  construct  validity,  internal 
validity  and  external  validity  (Yin,  2003:34)  Leedy  and  Ormrod  define  constructs  as 
“characteristics  that  cannot  be  directly  observed  but  must  instead  be  inferred  from 
patterns  in  people’s  behavior”  (Leedy  et  al,  2001:  98-99).  For  this  study,  an  effort  was 
made  to  ascertain  the  judgments  and  opinions  of  equipment  users  and  Automated  Test 
Equipment  experts  in  the  field  regarding  issues  of  supportability  and  efficiency.  These 
issues,  then,  are  the  constructs  for  this  study.  In  regards  to  construct  validity,  whenever 
possible,  data  were  incorporated  from  multiple  sources  of  evidence,  including  multiple 
interviews  from  different  personnel  and  historical  documents.  Conclusions  drawn  from 
the  collected  data  were  made  based  on  the  repetitive  claims  made  by  more  than  one 
respondent  from  a  given  program,  with  single-instance  occurrences  documented  as 
exceptions.  Additionally,  after  personal  face-to-face  or  telephone  interviews  were 
conducted,  respondents  were  supplied  with  a  copy  of  the  notes  from  the  interview.  This 
was  done  in  an  effort  to  verify  the  accuracy  of  collected  data. 

To  achieve  internal  validity,  using  the  data  collection  matrix  shown  in  Appendix  F, 
pattern-matching  and  categorical  aggregations  were  perfonned  to  cumulatively  assess  the 
data  obtained.  Likewise,  this  researcher  sought  to  achieve  convergence  of  fact  and, 
consequently,  external  validity,  by  targeting  responses  that  were  replicated  among 
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different  respondents.  These  responses  were  the  primary  basis  for  data  analysis  in 
chapter  four  as  well  as  theory  building  in  chapter  five. 

Finally,  in  regards  to  reliability,  Yin’s  (2003)  case  study  protocol  guidance  was 
used  as  a  means  to  structure  this  case  study.  The  protocol  helped  to  provide  a 
standardized  format  for  data  collection  and  transcription. 

Summary 

The  purpose  of  this  chapter  was  to  discuss  the  analysis  and  results  from  the  case 
study.  Similarities  and  differences  between  the  two  test  equipment  programs  under  study 
were  systematically  identified  and  discussed  in  the  order  established  by  the  case  study 
protocol.  We  then  addressed  the  validity  and  reliability  of  these  results  in  reference  to  the 
case  study  tactics  first  introduced  in  chapter  three. 

Next,  we  present  conclusions  from  the  research  in  an  attempt  to  address  the 
overarching  research  question  stated  at  the  beginning  of  this  study. 
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V.  Conclusions 


Chapter  Overview 

In  this  final  chapter,  additional  findings  are  presented  that  were  discovered  during 
the  course  of  this  research  that  were  not  explicitly  linked  to  the  study’s  investigative 
questions,  but  otherwise  present  interesting  and  important  implications  for  the 
management  of  F-15  test  equipment.  Next,  we  address  the  various  findings  as  they  relate 
to  the  overarching  research  question  posed  at  the  beginning  of  this  study.  Commentary 
on  research  limitations,  field  issues  and  data  collection  difficulties  are  discussed, 
followed  by  thoughts  on  related  areas  that  may  be  suitable  topics  for  future  research. 

Additional  Discoveries 

Several  additional  discoveries  were  made  during  the  course  of  this  research  and  are 
worth  consideration  when  evaluating  the  management  strategies  for  F-15  test  equipment. 
TISS  and  Historical  Organizational  Structure  Decisions 

When  reviewing  the  histories  of  both  pieces  of  test  equipment  observed  in  this 
study,  it  was  discovered  that  a  very  specific  managerial  action  was  taken  regarding  the 
commodity  consignment  of  TISS,  originally  scheduled  for  October  1997  (Raygor,  1996). 
Had  this  consignment  occurred,  the  focal  point  of  management  responsibility  for  TISS 
test  program  sets  would  have  been  split  apart  from  the  management  of  the  actual  test 
equipment  and  hardware.  Before  this  consignment  occurred,  a  proposal  was  internally 
submitted  and  approved  by  the  F-15  program  office  to  prevent  this  action  from 
happening.  This  proposal’s  recommendation  centered  on  the  fact  that  historically, 
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management  and  configuration  control  responsibility  for  TISS  had  resided  in  one  office 
at  the  Aeronautical  Systems  Center,  and  field  users  enjoyed  the  benefits  of  single  point  of 
contact  interface  with  program  management.  A  background  paper  included  as  part  of  this 
proposal  suggested  that  the  historical  arrangement  was  a  “validated  management 
system”,  and  to  reorganize  the  existing  management  structure  through  the  proposed 
commodity  consignment  would  be  to  jeopardize  the  level  of  support  for  field  users. 
Indeed,  the  proposal  suggested  that  “losing  the  TISS  single  point-of-contact  would  be 
disastrous  for  our  customers”  (Raygor,  1996). 

This  historical  document  was  found  to  be  supremely  relevant  to  this  case  study,  as 
the  importance  of  communication  and  successful  customer  interface  were  described  by 
respondents  as  essential  elements  of  managerial  support,  regardless  of  the  specific  test 
equipment  being  managed.  Furthermore,  it  was  ascertained  that  TISS  field  users 
continue  to  enjoy  the  benefits  of  sustained,  unchanging  lines  of  communication  with  the 
program  office  to  this  day. 

Continuity  and  Employee  Turnover 

An  additional  and  significant  difference  discovered  was  one  of  personnel  continuity 
and  employee  turnover.  Although  specific  data  regarding  turnover  was  not  collected  as 
part  of  the  case  study  protocol,  this  researcher  casually  observed  that  a  disparity  may 
exist  between  the  two  test  equipment  programs  in  terms  of  managerial  continuity.  In 
particular,  a  focal  manager  of  the  TISS  program  has  maintained  the  position  since  the 
equipment’s  inception  in  the  early  1990s.  By  contrast,  the  particular  position’s 
equivalent  counterpart  in  the  AIS  program  had  been  in  the  position  for  less  than  one  year. 
Additionally,  TISS  respondents  indicated  that  longstanding  employee  continuity  has  been 
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maintained  by  the  original  TISS  manufacturer.  Specifically,  a  core  group  of  engineers 
has  been  kept  in  place  by  the  manufacturer  since  TISS  was  first  developed,  and  these 
same  individuals  continue  to  work  closely  with  program  managers  today.  It  should  be 
noted  that  these  engineers  are  not  only  accepted  to  be  technical  experts  on  the  TISS 
system  itself,  but  also  on  the  supported  electronic  warfare  line-replaceable  units  from  the 
aircraft.  The  same  cannot  be  said  for  the  managers  currently  working  to  sustain  existing 
AIS  components  as  well  as  the  Antenna  Test  Station  and  Enhanced  Aircraft  Radar  Test 
Station.  The  Electronic  Systems  Test  Set,  the  intended  replacement  for  AIS,  was  not 
developed  by  the  same  manufacturer  as  the  original  AIS  system.  As  a  result,  there  is  no 
pre-existing  continuity  between  the  manufacturer  and  the  AIS  program  managers.  While 
the  specific,  measurable  impact  of  employee  continuity  on  the  level  of  managerial 
support  is  outside  the  scope  of  this  research,  it  seems  prudent  to  accept  that  higher 
continuity  on  a  given  program  will  lead  to  increased  expertise  regarding  its  workings.  At 
any  rate,  it  was  believed  that  this  topic  warranted  consideration  for  relevance  to  this 
study,  as  well  as  future  efforts. 

Roles  and  Responsibilities 

Most  respondents  were  eager  to  provide  commentary  and  feedback  regarding  the 
typical  roles  and  responsibilities  of  the  F-15  System  Program  Office  and  the  Automated 
Test  Systems  Division  (WR-ALC/LEA)  for  the  acquisition  and  life-cycle  management  of 
automated  test  equipment.  In  particular,  feedback  was  provided  on  the  commodity 
consignment  process  described  in  chapter  two.  As  a  review,  generally  speaking,  the 
acquisition  System  Program  Office  initiates  and  develops  a  given  piece  of  test  equipment 
upfront,  to  include  supporting  equipment,  spares,  technical  data  and  engineering 
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drawings  required  to  operate  and  subsequently  maintain  the  test  equipment  system.  By 
design,  an  integrated  product  team  is  formed  with  representatives  from  the  System 
Program  Office,  Major  Commands,  Air  Logistics  Centers  and  contractors  to  oversee  this 
process,  thereby  ensuring  adequate  support  from  “cradle  to  grave”.  Once  a  test 
equipment  system  enters  the  sustainment  phase,  practical  management  authority  is 
typically  consigned  away  from  the  System  Program  Office  to  the  Air  Logistics  Center. 
For  all  intents  and  purposes,  the  present  structure  of  TISS  is  an  aberration  -  overall 
management  has  not  been  consigned. 

Feedback  regarding  this  consignment  process  was  extremely  varied  among 
respondents.  Some  commented  that  consignment  is  a  necessary  reality  -  essentially 
saying  that  if  the  acquisition  personnel  never  relinquished  management  authority  they 
would  eventually  exhaust  all  capacity  and  lose  the  ability  to  take  on  and  develop  new 
projects.  This  certainly  seems  plausible,  but  then  again,  the  TISS  program  exists  as  an 
exception.  Other  comments  suggested  a  widespread  belief  that  different  and  better 
capabilities  exist  at  the  System  Program  Office  as  opposed  to  the  depot  -  specifically 
access  to  3010  and  3600  funds.  Additionally,  some  respondents  suggested  that  Air 
Logistics  Center  representatives,  while  involved  in  the  consignment  process,  are  not 
involved  enough  during  the  initial  acquisition  phase,  effectively  limiting  their  ability  to 
advocate  for  perceived  future  sustainment  needs  early  on  in  product  development. 
Another  concern  was  the  need  for  total  procurement  of  engineering  data  and  drawings. 

In  particular,  AIS  respondents  indicated  that  historical  funding  changes  resulted  in  a 
failure  to  obtain  this  data  upfront.  Presently,  AIS  managers  are  suffering  by  not 
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possessing  this  infonnation,  as  they  need  to  procure  follow-on  spares  or  replacements 
through  reverse  engineering  efforts  or  expensive  sole  source  acquisitions. 

Clearly,  there  are  many  viewpoints  and  opinions  regarding  the  commodity 
consignment  process  of  F-15  test  equipment.  This  researcher  believes  that  this  process 
alone  would  warrant  further  and  more  detailed  study.  In  the  next  section,  we  provide 
concluding  thoughts  and  commentary  on  the  results  of  this  study  and  address  the  original 
research  question. 

Addressing  the  Research  Question 

Is  F-15  automatic  test  equipment  more  supportable  and  efficient  when  managed  by 
a  separate  support  office  or  when  managed  by  the  supported  weapon  system? 

After  conducting  the  study,  it  has  been  detennined  that  sufficient  evidence  exists  to 
suggest  that  there  are  key  differences  between  the  two  F-15  automatic  test  equipment 
management  strategies  that  affect  the  supportability  and  efficiency  of  the  managed 
equipment.  The  completed  data  collection  matrix,  shown  in  Appendix  G,  was  evaluated 
and  assessed  in  conjunction  with  other  relevant  findings  presented  above  to  generate  the 
consolidated  figure  below.  Figure  14  identifies  which  of  the  two  test  equipment 
programs  was  determined  to  be  more  supportable  and  efficient  when  viewed  in  terms  of 
each  investigative  question. 
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Figure  14.  Summary  of  Protocol  Question  Findings 
This  researcher  wishes  to  emphasize  that  the  above  characterization  is  not  intended 
to  criticize  the  performance  of  any  individuals  within  the  AIS  program  -  indeed,  there  is 
no  such  claim  being  made.  Rather,  this  researcher  believes  that  there  are  certain  benefits 
to  be  gained  when  test  equipment  is  managed  primarily  by  the  supported  weapon  system. 
Investigative  questions  one  through  three  addressed  individual  management  approach, 
success  with  funding,  and  overall  health  of  each  program.  The  results  of  chapter  four 
indicate  that  TISS  has  traditionally  been  more  successful  in  each  of  these  areas. 
Investigative  question  four  did  not  yield  conclusive  results,  but  will  be  addressed  in  the 
limitations  and  field  issues  section  below. 

Clearly,  there  are  certain  characteristic  differences  between  the  two  test  equipment 
systems  that  have  an  impact  on  their  managers’  ability  to  support  them,  such  as  system 
age  and  different  supported  line-replaceable  units.  To  be  sure,  one  must  consider  that  we 
are  not  evaluating  the  same  system  managed  by  two  different  entities  (e.g.  comparing 
“apples  to  apples”). .  .indeed,  these  are  different  pieces  of  test  equipment,  and  that  fact 
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alone  has  its  implications.  That  being  said,  this  researcher  contends  that  TISS 
management  has  capitalized  on  the  benefits  of  collocation  with  the  weapon  system 
program  office.  Evidence  indicates  that  the  TISS  program  has  executed  sound  planning 
for  projected  equipment  obsolescence,  to  include  successfully  justifying  and  securing 
funding  in  the  competitive  program  objective  memorandum  process. 

Research  Limitations,  Field  Issues  and  Bias 

During  the  course  of  this  study,  several  limitations  and  issues  were  encountered  that 
could  have  either  introduced  bias  or  limited  the  scope  of  research.  This  section  will 
address  each  of  these  concerns: 

1.  Limitation:  Analysis  Limited  to  Two  Test  Equipment  Systems 

First  and  foremost,  this  case  study  analysis  was  limited  to  an  examination  of  only 
two  test  equipment  programs  within  the  F-15  community:  TISS  and  AIS.  The  results  of 
this  study  are  therefore  limited  only  to  these  two  systems;  similar  assessments  of  other 
test  equipment  programs  in  other  weapon  systems  should  be  made  before  making  broader 
assumptions  regarding  managerial  strategies. 

2.  Field  Issue:  Budget  and  Expenditure  Data 

This  researcher  was  only  able  to  obtain  historical  funding  data  for  the  past  five 
fiscal  years.  With  a  greater  volume  of  data,  a  more  robust  analysis  could  be  performed  in 
regards  to  funding  trend  determinations,  to  include  possible  statistical  testing. 
Additionally,  this  researcher  encountered  difficulty  in  obtaining  the  historical  funding 
dollar  values  through  military  archives  or  records.  That  is,  other  individuals  from  the 
respective  test  equipment  programs  were  relied  upon  to  provide  the  relevant  funding  data. 
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As  a  consequence,  this  researcher  was  not  in  a  position  to  verify  the  accuracy  of  the 
figures  presented. 

3.  Field  Issue:  Encountering  Reluctance  on  the  Part  of  Respondents 

Particularly  in  regards  to  data  collection  for  Investigative  Question  four,  this 

researcher  sensed  a  general  hesitance  on  the  part  of  respondents  to  speak  candidly  about 
the  perfonnance  of  both  the  program  office  and  depot  in  terms  of  supporting  field  units. 
This  researcher  did  undergo  the  necessary  protocol  to  ensure  anonymity  and  non¬ 
retribution,  but  was  only  able  to  ascertain  limited  feedback  pertaining  to  this  investigative 
question.  In  hindsight,  this  researcher  suspects  that  due  to  the  tight-knit  nature  of  the 
weapon  system  community,  respondents  were  reluctant  to  offer  comments  that  may  have 
been  construed  (or  misconstrued)  to  be  critical  of  others. 

4.  Limitation  and  Potential  Bias:  Data  Collection 

It  should  be  noted  that  there  was  a  disparity  in  the  means  of  data  collection  from  all 
respondents.  This  researcher  was  able  to  interview  and  collect  data  from  the  TISS 
program  office  in  person,  being  located  on  the  same  installation.  Other  respondents  at 
separate  geographic  locations  were  interviewed  both  by  telephone  and  through  electronic 
communication.  As  a  result,  bias  could  have  been  introduced  due  to  the  ability  to 
observe  non-verbal  communication  in  face-to-face  discussions. 

5.  Limitation  and  Potential  Bias:  Researcher  Experience 

This  researcher,  while  having  prior  maintenance  experience  in  the  Air  Force,  had 
no  previous  experience  working  within  the  F-15  weapon  system  community.  While  this 
fact  alone  suggests  that  this  researcher  would  be  an  apropos  choice  in  terms  of 
objectivity,  this  researcher  did  have  experience  working  with  test  equipment  and 
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obsolescence,  albeit  under  a  different  weapon  system  platform.  As  a  result,  this 
researcher  had  prior  knowledge  and  experience  with  the  Automated  Test  Systems 
Division;  these  experiences  may  have  introduced  preconceived  notions  and  bias  into  this 
research. 

Recommendations  for  Future  Research 

While  conducting  this  study,  three  areas  were  noted  that  may  be  suitable  candidates 
for  future  topics  of  research.  First,  as  stated  previously,  this  research  was  conducted  in 
part  to  serve  as  a  foundation  and  baseline  template  for  similar  inquiries  into  the 
management  strategies  for  test  equipment  systems  employed  by  other  weapon  systems. 
This  researcher  developed  the  case  study  protocol  for  this  effort  in  tandem  with  another 
researcher  examining  strategic  missile  test  equipment  (Ford,  2005).  It  is  certainly 
plausible,  therefore,  that  future  researchers  may  build  upon  the  structure  presented  in  this 
study  and  pursue  separate  inquiries  into  the  management  of  other  test  equipment 
programs.  Next,  during  the  course  of  this  study,  it  was  determined  that  it  would  be  useful 
to  build  a  separate  comparison  study  between  similar  test  equipment  systems  using 
reliability,  maintainability  and  life-cycle  costing  techniques.  For  this  study,  the  protocol 
did  not  explicitly  request  historical  reliability  data,  but  such  an  endeavor  certainly  seems 
feasible.  Moreover,  TISS  and  AIS  are  used  to  test  very  different  line-replaceable  units;  a 
reliability  and  maintainability  study  may  be  better  suited  for  comparing  two  test 
equipment  systems  that  support  similar  line-replaceable  units  for  different  weapon 
systems.  Finally,  it  may  be  worthwhile  to  critically  evaluate  the  oft-encountered  choice 
between  staying  with  the  original  equipment  manufacturer  for  all  future  sustainment 
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needs  versus  opening  up  repair,  support  or  next-generation  contracts  to  new  providers. 

On  a  topical  level,  it  is  possible  that  there  may  be  a  positive  correlation  between  long¬ 
term  sustainability  and  uninterrupted  support  from  the  original  equipment  manufacturer. 

Research  Summary 

The  purpose  of  this  research  was  to  examine  the  implications  of  two  alternative 
strategies  regarding  the  management  of  automated  test  equipment:  single-point 
management  under  the  weapon  system  ownership  of  a  System  Program  Office,  or 
separate  management  at  an  existing  stand-alone  automated  test  systems  support  office. 
Using  a  case  study  methodology,  we  developed  a  protocol  and  collected  information  on 
the  management  of  two  separate  test  equipment  systems  used  to  support  the  F-15 
platform:  the  Tactical  Electronic  Warfare  System  Intermediate  Service  Station  (TISS), 
which  is  primarily  managed  at  the  F-15  System  Program  Office,  and  the  Avionics 
Intermediate  Shop  (AIS),  which  is  managed  at  the  Automated  Test  Systems  Division  at 
Warner-Robins  Air  Logistics  Center.  Results  of  the  research  suggest  that  the  equipment 
managed  by  the  system  program  office  (TISS)  has  historically  been  more  supportable  and 
efficient  than  the  equipment  managed  by  the  stand-alone  automated  test  systems  support 
office  (AIS). 

Recommendation 

Consider  the  initial  charge  found  within  Air  Force  Policy  Directive  63-2,  previously 
introduced  in  chapter  one  of  this  thesis:  “Standardized  automatic  test  systems  (ATS)  and 
equipment  can  provide  efficiency  and  reduced  cost  by  minimizing  the  proliferation  of 
system-unique  test  equipment  while  ensuring  that  the  maintenance  and  deployment 
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requirements  of  existing  and  developing  weapon  systems  and  equipment  are  met." 
(USAF-1,  1994).  If  we  are  to  accept  common  test  equipment  as  a  viable  solution  for 
future  test  equipment  needs,  managerial  support  for  that  equipment  must  be  responsive 
and  non-hindering  to  the  maintenance  abilities  of  supported  aircraft  and  weapon  systems. 
Obsolete,  ill-funded  test  equipment  such  as  the  Avionics  Intermediate  Shop  has  hampered 
its  managers’  ability  to  meet  this  endeavor.  While  all  efforts  are  made  to  make  the  best 
use  of  scarce  resources,  sacrificing  the  level  of  support  for  product  end-users  should  be 
avoided  if  at  all  possible.  What  then,  can  be  done  to  improve  the  viability  of  the  current 
approach  of  consolidating  test  equipment  sustaimnent  management  at  a  stand-alone 
office? 

First,  there  needs  to  be  a  more  rigorous,  dedicated  effort  to  providing  adequate 
funding  for  test  equipment  in  the  post-acquisition  phase.  The  AIS  program  is 
representative  of  numerous  testers  in  sustainment  at  the  Automated  Test  Systems 
Division  that  suffer  due  to  repeated  partial  funding  of  requirements.  The  TISS  program 
has  benefited  from  access  to  production  and  development  funds  that  are  only  available  at 
the  System  Program  Office;  the  depot  is  not  privy  to  3600  and  3010  monies.  This  is  one 
factor  that  has  enabled  TISS  managers  to  better  confront  the  problems  of  increased 
obsolescence.  When  weapon  system  service  lives  are  extended,  the  test  equipment  used 
to  support  those  weapon  systems  must  be  correspondingly  extended,  upgraded  or 
replaced.  If  projected  improvement  funds  are  not  budgeted  for  in  the  acquisition  phase, 
the  funds  must  be  provided  to  the  management  programs  in  the  sustainment  phase. 

Next,  the  Automated  Test  Systems  Division  must  be  more  actively  involved  in  the 
acquisition  and  development  process  for  next-generation  or  replacement  testers.  Initial 
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findings  in  the  2003  General  Accounting  Office  report  suggested  that  depot  personnel  are 
not  involved  to  the  level  they  should  be  in  order  to  address  essential  components  of  future 
sustainment  for  a  new  system.  As  an  example,  depot  personnel  indicated  that  the  AIS 
program  has  suffered  today  as  a  result  of  not  procuring  certain  maintenance  specifications 
and  engineering  drawings  during  the  acquisition  phase;  this  information  would  have  been 
extremely  valuable  in  terms  of  obtaining  replacement  spares  or  obsolete  components. 
Dwindling  production  lines  and  manufacturing  sources  complicate  this  problem,  and  the 
lack  of  internally  owned  engineering  documentation  from  the  original  manufacturer  can 
result  in  less  than  desirable  reverse  engineering  efforts  or  exorbitant  sole-source 
purchases.  Moreover,  the  Automated  Test  Systems  Division,  as  the  Air  Force’s  Product 
Group  Manager  has  been  charged  with  a  degree  of  oversight  in  the  acquisition  process, 
both  to  minimize  the  proliferation  of  unique  testers  and  capitalize  on  opportunities  for 
common  capability  across  weapon  systems.  If  System  Program  Offices  do  not  involve 
the  Automated  Test  Systems  Division  in  their  acquisition  decision-making,  any 
anticipated  benefits  espoused  by  service-level  policy  are  likely  to  go  unrealized. 

Finally,  Major  Command  leadership  should  consider  the  establishment  of  a  liaison 
position  at  the  Automated  Test  Systems  Division  to  serve  as  an  advocate  for  managed  test 
equipment  in  the  sustainment  phase.  Often,  this  liaison  is  in  close  communication  with 
field  users  regarding  the  day-to-day  challenges  encountered  in  maintenance  units.  A 
depot-placed  experienced  career  technician  who  is  well-versed  in  the  particular  test 
equipment  could  perhaps  serve  as  a  more  effective  lobbyist  to  Major  Command  decision¬ 
makers.  This  position  has  already  been  established  at  the  System  Program  Office  to 
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monitor  the  development  of  products  in  the  acquisition  phase;  it  seems  conceivable  that 
benefits  would  be  gained  from  establishing  a  mirror  position  at  the  depot. 

In  summary,  this  research  suggests  that  test  equipment  programs  managed  by  their 
program  offices  through  sustainment  are  more  successful  than  those  which  are  distanced 
from  the  supported  platform  and  consigned  to  a  stand-alone  office.  Although  existing  Air 
Force  policy  advocates  against  the  proliferation  of  unique  test  equipment  and  espouses 
perceived  benefits  of  consolidated,  common  test  equipment,  any  decision  to  separate 
formal  management  responsibility  from  the  owning  weapon  system  platform  must  take 
into  consideration  the  other  ramifications  associated  with  taking  such  action. 
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Appendix  A:  IQ  1  Case  Study  Protocol 


Investigative  Question  #1 :  What  are  the  differences  in  how  ATE  is  being  managed? 

Sources  of  Data: 

SPO  and  Depot 

Program  Managers 

Equipment  Specialists 

Organizational  Charts 

Supporting  Questions: 

(Please  type  answers  directly  below  each  question  and  use  as  much  space  as  required) 

1 .  What  is  your  duty  title  and  job  description? 

2.  What  activities  are  part  of  your  daily  work  that  pertain  to  the  management  of  this 
equipment? 

3.  What  activities  do  you  perfonn  that  pertain  to  short-,  mid-  and  long-term  planning  for 
this  equipment? 

4.  What  activities  do  you  perfonn  in  support  of  or  in  response  to  issues  that  arise  from 
field-level  users  of  this  equipment? 

5.  When  was  this  equipment  initially  fielded? 

6.  What  did  this  equipment  replace,  if  applicable? 

7.  Have  you  encountered  any  obsolescence  issues  pertaining  to  this  equipment?  If  so, 
please  explain. 

8.  What  managerial  challenges  do  you  perceive  as  unique  to  this  equipment? 

9.  With  whom  do  you  routinely  work  in  support  of  managing  this  equipment?  (duty 
titles  and  job  descriptions) 
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10.  Are  you  familiar  with  DoD  policy  regarding  the  acquisition  and  development  of 
common  test  equipment?  (AFPD  63-2) _ 


1 1 .  How  does  this  DoD  policy  affect  this  equipment,  if  at  all? 


12.  Please  add  any  additional  infonnation  you  feel  would  aid  in  answering  Investigative 
Question  #1. _ 
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Appendix  B:  IQ  2  Case  Study  Protocol 


Investigative  Question  #2:  How  much  funding  is  budgeted/funded  in  the  Program 
Objective  Memorandum  (POM)  and  Future  Year  Defense  Plan  (FYDP)  for  your  ATS 
program? 

Sources  of  Data: 

MAJCOM  and  SPO/depot  financial  managers 

MAJCOM  and  SPO/depot  subject  matter  experts 

Budget  reports 

Budget  Estimate  Submissions  (BES) 

Supporting  Questions: 

(Please  type  answers  directly  below  each  question  and  use  as  much  space  as  required) 

1 .  Who  is  responsible  for  funding  your  related  equipment? 

2.  Are  there  any  funded/unfunded  requirements  for  the  equipment?  Please  explain. 

3.  What  equipment  funding  requirements  were  included  in  the  last  2  POM  cycles? 

Please  include  the  associated  BES  input. 

4.  Do  you  get  the  funds  needed  to  adequately  support  your  equipment  program?  Please 
explain. 

5.  Are  your  equipment  POM  inputs  funded?  If  not,  how  far  below  the  Air 

Staff/MAJCOM  funding  line  did  the  requirements  fall? 

6.  Please  add  any  additional  infonnation  you  feel  would  aid  in  answering  Investigative 
Question  #2. 
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Appendix  C:  IQ  3  Case  Study  Protocol 


Investigative  Question  #3:  What  are  ATE  SPO,  depot,  and  MAJCOM  assessments  of 
ATE/ATS  sustainability  for  the  two  programs? 

Sources  of  Data: 

MAJCOM  Staff 

SPO/depot  program  managers 

SPO/depot  equipment  specialists 

Supporting  Questions: 

(Please  type  answers  directly  below  each  question  and  use  as  much  space  as  required) 

1.  What  specific  sustainability  issues  (e.g.  hardware,  components)  does  this  equipment 
have? 

2.  Is  there  an  action  plan  to  address  these  issues?  Please  explain. 

3.  Are  these  issues  solvable?  If  not,  why?  What  roadblocks,  if  any,  are  hindrances  to 
reaching  a  solution? 

4.  Are  there  timelines  for  reaching  a  solution?  Please  explain. 

5.  When  did  these  issues  first  arise? 

6.  How  urgent  of  a  priority  are  these  issues? 

7.  Please  add  any  additional  infonnation  you  feel  would  aid  in  answering  Investigative 
Question  #3. 
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Appendix  D:  IQ  4  Case  Study  Protocol 


Investigative  Question  #4:  What  are  MAJCOM  assessments  of  field  units’  abilities  to 
support  assigned  equipment  with  resources  provided  by  SPO/depot? 

Sources  of  Data: 

MAJCOM  subject  matter  experts 

Memorandums  documenting  ATE/ATS  problems 

Supporting  Questions: 

(Please  type  answers  directly  below  each  question  and  use  as  much  space  as  required) 

1 .  Do  units  get  the  parts  they  need  to  support  their  assigned  equipment?  Please  explain. 

2.  Do  units  get  the  replacement  parts  in  a  timely  fashion?  Please  explain. 

3.  How  often  do  technicians  have  to  rely  on  technical  procedures  (at  the  direction  of  the 
SPO)  not  provided  in  established  TO  procedures?  Who  proposed  the  solution  (field, 
MAJCOM,  or  SPO/depot)?  Please  explain. 

4.  Are  TOs  of  good  quality?  Please  explain. 

5.  Are  TCTOs  of  good  quality?  Please  explain. 

6.  Do  units  receive  adequate  SPO  troubleshooting  support  when  formal  technical  data  is 
exhausted?  Please  explain. 

7.  Are  there  relatively  more,  fewer,  or  about  the  same  number  of  MICAPs  as  in  the  past? 
Please  explain. 

8.  Are  backorders  longer  than  30  days  common?  Give  examples  of  recurring  problems. 

9.  Please  add  any  additional  infonnation  you  feel  would  aid  in  answering  Investigative 
Question  #4. 
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Appendix  E:  Historical  Sustainment  Funding  for  TISS  and  AIS,  FY01-FY05 


AIS 

583 

MSD  Eng 

MSD  UUT 

540 

545 

SUM 

FY01 

$ 

$  6,100,000.00 

$  9,400,000.00 

$ 

$ 

$  15,500,000.00 

FY02 

$ 

$  6,100,000.00 

$ 

$  118,000.00 

$ 

$  6,218,000.00 

FY03 

$ 

$  6,600,000.00 

$ 

$  1,600,000.00 

$ 

$  8,200,000.00 

FY04 

$  933,000.00 

$  4,200,000.00 

$ 

$  1,400,000.00 

$463,000.00 

$  6,996,000.00 

FY05 

$  6,000,000.00 

$  14,000,000.00 

$  7,000,000.00 

$  2,000,000.00 

$300,000.00 

$  29,300,000.00 

SUM 

$  6,933,000.00 

$  37,000,000.00 

$  16,400,000.00 

$  5,118,000.00 

$763,000.00 

$  66,214,000.00 

TISS 

583 

MSD  Eng 

MSD  UUT 

540 

545 

Repairs 

Spares 

SUM 

FY01 

$  600,000.00 

$ 

$ 

$ 

$ 

$  3,120,000.00 

$  350,000.00 

$  4,070,000.00 

FY02 

$  600,000.00 

$ 

$ 

$ 

$ 

$  3,120,000.00 

$  350,000.00 

$  4,070,000.00 

FY03 

$  600,000.00 

$  1,030,000.00 

$ 

$ 

$ 

$  3,120,000.00 

$  350,000.00 

$  5,100,000.00 

FY04 

$  635,000.00 

$  1,030,000.00 

$ 

$ 

$ 

$  3,192,000.00 

$  723,000.00 

$  5,580,000.00 

FY05 

$  635,000.00 

$  3,550,000.00 

$ 

$ 

$ 

$  3,320,000.00 

$  370,000.00 

$  7,875,000.00 

SUM 

$  3,070,000.00 

$  5,610,000.00 

$ 

$ 

$ 

$  15,872,000.00 

$  2,143,000.00 

$  26,695,000.00 
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Appendix  F:  Data  Collection  Matrix  Template 
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Appendix  G:  Completed  Data  Collection  Matrix 


Type  and  Quantity  of  Data 


Compare  Comment/Theme/Note/Trend 


Each  Question 


Similar  structure  of  Integrated  Product  Teams;  Logistics, 
Engineering,  Contracting,  Tech  Orders;  scope  difference 


Similar  coordinating  activities 


Possible  difference  in  field-user  input;  contractor  involvement 


T1SS:  collaborative  efforts  with  IPT,  contractor;  AIS:  independent 
problem-solving 


TISS  initially  fielded  1992  vice  AIS  early  1970s 


TISS  replaced  TITE;  AIS  is  first  generation 


AIS  suffered  more  due  to  obsolescence  than  TISS 


TISS  staff  focused  on  complexity  of  TEWS;  AIS  staff  focused  o 
obsolescence,  parts  procurement 


Similar  staff  personnel,  MAJCOM  and  contractor  involvement 


minimal  familiarity;  general  distaste  for  policy 


N/A  for  current  testers  (in  sustainment);  general  feelings  against 


TISS  receipt  of 3600/3010  funds 


TTIP;  impact  of  ESTS 


Greater  obsolescence  for  AIS 


Increased  MICAPs  due  to  ops  tempo 


Varies:  less  than  30  to  120  days 


Appendix  H:  List  of  Terms  and  Commonly  Used  Acronyms 


AFB 

Air  Force  Base 

AFC ATE 

Air  Force  Common  Automatic  Test  Equipment 

AFI 

Air  Force  Instruction 

AFMC 

Air  Force  Materiel  Command 

AFPD 

Air  Force  Policy  Directive 

AFRL 

Air  Force  Research  Laboratory 

AIS 

Avionics  Intennediate  Shop 

ALC 

Air  Logistics  Center 

ARTS 

Aircraft  Radar  Test  Station 

ASC 

Aeronautical  Systems  Center 

ATE 

Automated  Test  Equipment 

ATS 

Antenna  Test  Station 

ATS 

Automated  Test  Systems 

ATS/E 

Automated  Test  Systems/Equipment 

AUR 

All-up  Round 

AWC  BBS 

Air  Warfare  Center  Bulletin  Board  System 

CJ 

Commodity  Class  Consignment 

CNI 

Communication,  Navigation  and  Identification 

COTS 

Commercial  Off-the-shelf 

CTS 

Computer  Test  Station 

DMSMS 

Diminishing  Manufacturing  Sources  and  Material  Shortages 

DoD 

Department  of  Defense 

DoDI 

Department  of  Defense  Instruction 

DTS 

Display  Test  Station 

EA 

Executive  Agent 

EARTS 

Enhanced  Aircraft  Radar  Test  Station 

ED 

Executive  Directorate 

ESTS 

Electronic  Systems  Test  Set 

FYDP 

Future  Year  Defense  Plan 

GAO 

General  Accounting  Office 

HAF 

Headquarters  Air  Force 

HQ 

Headquarters 

I&  C 

Indicators  and  Controls 

IPT 

Integrated  Product  Team 

IWSM 

Integrated  Weapon  System  Management 

LRU 

Line  Replaceable  Unit 

MAJCOM 

Major  Command 

METS 

Mobile  Electronic  Test  Set 

MICAP 

Mission  Capability  (Highest  Supply  Demand  Priority) 

MOA 

Memorandum  of  Agreement 

MSD 

Material  Support  Division 
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MTS 

Microwave  Test  Station 

OI 

Operating  Instruction 

OSD 

Office  of  the  Secretary  of  Defense 

PGM 

Product  Group  Manager 

PMRT 

Program  Management  Responsibility  Transfer 

POM 

Program  Objective  Memorandum 

PM 

Program  Manager 

SAF 

Secretary  of  the  Air  Force 

SPD 

System  Program  Director 

SPO 

System  Program  Office 

SRU 

Shop  Replaceable  Unit 

TCTO 

Time  Compliance  Technical  Order 

TEWS 

Tactical  Electronic  Warfare  System 

TISS 

TEWS  Intermediate  Service  Station 

TITE 

TEWS  Intermediate  Test  Equipment 

TO 

Technical  Order 

TPS 

Test  Program  Set 

TRU 

Tester  Replaceable  Unit 

TTIP 

TISS  Technology  Insertion  Program 

USAF 

United  States  Air  Force 

UUT 

Unit-under-test 

VXI 

Versa  Module  Eurocard  Extensions  for  Industry 

WPAFB 

Wright-Patterson  Air  Force  Base,  OH 

WR-ALC 

Warner-Robins  Air  Logistics  Center,  GA 

WR-ALC/LEA 

WR-ALC  ATS  Division;  Air  Force  ATS  PGM 
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Appendix  I:  Selected  SPO  Data  for  IQ  1 


Investigative  Question  #1 :  What  are  the  differences  in  how  ATE  is  being  managed? 


Sources  of  Data: 

SPO  and  Depot 

Program  Managers 
Equipment  Specialists 
Organizational  Charts 

Supporting  Questions: 

(Please  type  answers  directly  below  each  question  and  use  as  much  space  as  required) 


1 .  What  is  your  duty  title  and  job  description? 


-  DUTY  TITLE  MASKED 

o  Performs  LM  &  PM  functions 
o  Coordinates  parts  problems  (bases  &  contractor) 
o  Management  side  -  planning,  TTIP  program,  modernization 

-  DUTY  TITLE  MASKED 

o  Interface  with  DRA 1 5 
o  Monitor  progress  on  ACC’s  requirements 
o  “watchdog”  for  ACC  interests 

Overall  management  of  the  TISS  Program  to  include  funding  all  modifications 
(software  and  hardware),  T.O.s,  Field  Engineering  support  and  Configuration 
Management  of  the  system 

-  DUTY  TITLE  MASKED 


2.  What  activities  are  part  of  your  daily  work  that  pertain  to  the  management  of  this 
equipment? _ 

coordinate  with  MANUFACTURER  NAME  MASKED ,  WR,  field  sites  on  parts 
problems  (shortages)  and  obsolescence  issues 

run  TCMs  -  bimonthly  meetings  with  all  involved  parties  (essentially  an  IPT) 
TTIP  (TISS  technology  insertion  program)  upgrade  of  COTS  equipment;  GP  and 
RF  interfaces 

Coordinate  base/contractor  issues 
-  Working  08-11  POM 


3.  What  activities  do  you  perform  that  pertain  to  short-,  mid-  and  long-term  planning  for 
this  equipment? _ 


Work  closely  with  MANUFACTURER  NAME  MASKED  representatives  on  all 
ranges  of  planning 
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Actively  involve  people  from  field  in  planning  process 

Emphasize  the  importance  of  frequent  user  input  throughout  planning  and 

development 

Engineering  support  budgeted  for 

Meet  every  other  month;  comprehensive  program  website  accessible  by  all  users 
and  members  of  the  program  community 
Website  components  include: 
o  TISS  General  Information 
o  TISS  Logistics  Information 
o  TISS  Technology  Insertion  Program 
o  TISS  Commercial  Computer  Support 
o  AWC  BBS  Information 
o  TISS  Software  Configuration 
o  TISS  Deficiency  Reports  (DR) 
o  TISS  Field  Bulletins 
o  TISS  Field  Problem  Reports 
o  TISS  Technical  Coordination  Meetings  (TCM) 
o  TISS  SERD  List/TISS  Allowance  Standard 
o  TISS  T.O.  List 
o  TISS  Workaround  Summary 
o  TISS  Silver  Bullets 
o  TISS  Training 

Users  perspective  genuinely  sought  (multiple  respondents  re-iterated) 


4.  What  activities  do  you  perform  in  support  of  or  in  response  to  issues  that  arise  from 
field-level  users  of  this  equipment? _ 


IPT/TCM  meet  to  review  issues  together  (every  other  month) 

o  SPO,  ACC,  LEACC,  LSRAC,  MANUFACTURER  NAME  MASKED, 
MANUFACTURER  NAME  MASKED,  MANUFACTURER  NAME 
MASKED,  Schoolhouse 

Have  capability  to  use  AWC  BBS  for  classified  software  updates 
o  Total  replacement  of  software  as  opposed  to  patches 
o  Field  users  are  able  to  rapidly  download  all  software  updates  to  their  test 
stations 

o  This  is  a  classified  system  used  by  AWC  and  WR-ALC/LS  (Electronic 
Warfare  Division)  to  send  out  classified  Aircraft  flight  programs.  All  the 
F-15  shops  have  access  to  it.  We  were  able  to  take  advantage  of  this 
because  most  of  our  TISS  Software  is  classified. 

Coordinate  field  issues  with  contractor  (getting  parts  where  they  are  needed, 
getting  help  on  site) 


5.  When  was  this  equipment  initially  fielded? 
1992,  development  90-92 _ 
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NOTE:  reference  TISS  historical  timeline  on  program  website 

TISS  remained  at  SPO  vs.  LEA  PMRT  (as  well  as  assoc.  TPS) 

o  SSS;  received  formal  approval  to  keep  focal  point  of  management  as  is 
instead  of  splitting  up  TPS/ATS  ownership  and  transferring  everything 
down  to  LEA  (have  a  copy  of  this  report) 


6.  What  did  this  equipment  replace,  if  applicable? 


TITE  was  old  system 

Originally  intended  for  ECP  to  support  new  TEWS  ($65M) 

Instead,  competed  a  new  system  for  $200M 
SPO  director  decision 

Initially  had  to  use  MATE  computer  -  costly  and  not  friendly  to  use  (multiple 
respondents  expressed  distaste  for  MATE) 


7.  Have  you  encountered  any  obsolescence  issues  pertaining  to  this  equipment?  If  so, 
please  explain. _ 


COTS  equipment  from  the  mid  ‘80s 

Just  now  starting  to  have  some  real  problems,  parts  shortages 
Performed  a  case  study  internally  in  ’02  predicting  projected  shortages  in  ’07  & 
’08  (losing  capability  of  stations)  -  believe  case  study  was  a  big  help  to  funding 
approval  for  TTIP 

SPO  has  a  great  deal  of  flexibility  on  workarounds 

Formal  workaround  process  signed  off  by  ACC/  MANUFACTURER  NAME 
MASKED  /SPO;  managed  at  SPO 

Internally  managed  website  that  all  have  access  to  for  posting  of  workarounds, 
communication 


8.  What  managerial  challenges  do  you  perceive  as  unique  to  this  equipment? 


TEWS  are  some  of  the  most  complex  LRUs 

Didn’t  transfer  to  LEA  (reference  copy  of  approved  request) 


9.  With  whom  do  you  routinely  work  in  support  of  managing  this  equipment?  (duty 
titles  and  job  descriptions) _ 


Reference  POC  list  provided 
Reference  internal  TISS  website 


10.  Are  you  familiar  with  DoD  policy  regarding  the  acquisition  and  development  of 
common  test  equipment?  (AFPD  63-2) 


Yes,  doesn’t  apply  because  this  piece  of  equipment  is  in  sustainment  phase 
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1 1 .  How  does  this  DoD  policy  affect  this  equipment,  if  at  all? 


It  would  be  “overkill”  to  make  this  piece  of  equipment  common 
TEWS  is  too  complicated  to  generalize 

Common  ATS  will  only  be  successful  if  aircraft  have  common  LRUs 


12.  Please  add  any  additional  infonnation  you  feel  would  aid  in  answering  Investigative 
Question  #1. _ 
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Appendix  J:  Selected  Depot  Data  for  IQ  1 


Investigative  Question  #1 :  What  are  the  differences  in  how  ATE  is  being  managed? 


Sources  of  Data: 

SPO  and  Depot 

Program  Managers 
Equipment  Specialists 
Organizational  Charts 

Supporting  Questions: 

(Please  type  answers  directly  below  each  question  and  use  as  much  space  as  required) 


1 .  What  is  your  duty  title  and  job  description? 


DUTY  TITLE  MASKED.  I  am  responsible  for  the  F-15  AIS  program. 
DUTY  TITLE  MASKED 
DUTY  TITLE  MASKED 


2.  What  activities  are  part  of  your  daily  work  that  pertain  to  the  management  of  this 
equipment? _ 


Work  with  the  IPT  team  ( DUTY  TITLES  MASKED)  together  we  manage  the  AIS 
program 

WR-ALC/LEA’s  responsibility  is  to  sustain  the  weapon  system  ATE  for  the  life  of  the 
system.  The  SPO  acquires  the  ATE  upfront  and  all  supporting  equipment/spares/tech 
data/engineering  drawings/training/etc.  required  to  operate  and  maintain  the  ATE. 

During  the  acquisition  process  the  SPO,  MAJCOMs,  ALCs,  and  contractors  fonn  an  IPT 
to  ensure  the  ATE  can  be  supported  from  cradle  to  the  grave.  The  driving  force  behind 
the  functional  requirements  of  this  IPT  is  the  mission  statement  specified  in  the 
Operational  Requirements  Document  (ORD)  and  Mission  Needs  Statement  (MNS)  which 
is  prepared  by  the  users/MAJCOM 

Participation  by  all  members  is  critical  to  successful  fielding  and  Sustainment  of  ATE. 
The  MAJCOMs  provide  valuable  insight  into  the  day  to  day  functional  test  requirements. 
Every  effort  should  be  made  to  have  them  during  the  initial  and  all  follow-on  phases  of 
acquisition  and  sustainment. 


3.  What  activities  do  you  perfonn  that  pertain  to  short-,  mid-  and  long-term  planning  for 
this  equipment? _ 
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Since  this  program  is  over  30  yrs  old  it  is  hard  to  do  short  tenn  planning,  Mid  to  Long 
tenn  working  with  the  IMs  and  ESs  we  tried  to  predict  what  we  will  need  and  see  if  the 
item  is  in  the  warehouse  from  test  station  that  have  been  turned  in. 


4.  What  activities  do  you  perform  in  support  of  or  in  response  to  issues  that  arise  from 
field-level  users  of  this  equipment? _ 


When  issues  arise  from  the  field  the  ES  or  IM  will  work  the  issue  and  coordinate  with  me 
on  the  decision 


5.  When  was  this  equipment  initially  fielded? 
Early  1970s. 


6.  What  did  this  equipment  replace,  if  applicable? 
N/A. 


7.  Have  you  encountered  any  obsolescence  issues  pertaining  to  this  equipment?  If  so, 
please  explain. _ 


Due  to  the  age  of  the  system  (1960s/1970s)  technology,  the  AIS  has  numerous 
obsolescence  issues.  The  CTS,  DTS,  MTS,  CNI,  I&C  are  being  replaced  by  the  ESTS, 
but  the  ATS  and  EARTS  must  be  sustained  until  2025.  We  have  an  ongoing  upgrade 
program  with  Honeywell  to  develop  prototype  replacements  for  obsolete  tester  replace 
units  in  the  AIS. 

Obsolescence  (Multiple  respondents  indicated) 

~  50%  of  ARTS  Test  Replaceable  Units  (TRUs)  are  obsolete/non-procurable 
~  62%  of  MSIP  and  EARTS  TRUs  are  obsolete/non-procurable 

Diminishing  Manufactory  Sources 
Very  limited  repair  sources 

Repair  turnaround  time  exceeds  120  days  on  average 

I-level  experiencing  throughput  issues  due  to  parts  non-availability 

Adversely  impacts  Weapon  System  availability 

TRUs  spares  levels  50%  below  authorizations 

Cannibalization  virtually  non-existent  at  the  I-level/Depot 
Station  Mission  Capable  rates  rapidly  degrading  across  CAF 

Reference  F-15  ATS  Slideshow  included  -AUTHOR  NAME  MASKED. 


8.  What  managerial  challenges  do  you  perceive  as  unique  to  this  equipment? 
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Old  technology  generates  major  obsolescence  issues.  New  technology  insertions  lead  to 
incompatibility  issues  due  to  timing,  signal  processing,  and  theory  of  operation  changes. 
Lack  of  a  complete  engineering  data  package  leads  to  difficulty  in  procuring  spares  as  the 
equipment  becomes  obsolete  or  unsupportable. 

Adequate  support  equipment,  tools,  data,  and  training  must  be  procured  upfront,  to  ensure 
successful  Sustainment  of  the  ATE  for  the  life  of  the  system. 

All  these  tools  must  be  acquired  to  keep  the  ATE  operational  and  supportable.  The  ALC 
must  be  involved  in  the  acquisition  process  at  the  initial  stage  to  ensure  all  required 
essential  elements  are  addressed  to  sustain  the  equipment  for  the  life  of  the  system. 
Unfortunately  the  ALC  is  not  always  a  major  player  during  the  acquisition  phase.  Also 
budgetary  constraints  have  sometimes  lead  to  cutting  maintenance  data  such  as 
engineering  drawings  which  later  on  become  essential  for  procurement  of  follow-on 
spares  or  replacement  of  the  obsolete  ATE.  This  happened  with  the  management  of  the 
F-15  AIS  and  lead  to  both  risky  reverse  engineering  projects  and  expensive  sole  source 
acquisitions.  The  AIS  is  an  aging  system  that  contains  numerous  obsolescence  issues. 

To  ensure  the  user’s  mission  capability  is  not  degraded  and  the  ATE  can  be  acquired  at 
the  best  cost,  complete  engineering  data  should  always  be  procured  during  the  acquisition 
phase. 


9.  With  whom  do  you  routinely  work  in  support  of  managing  this  equipment?  (duty 
titles  and  job  descriptions) _ 


Command  HQ  ACC,  Equipment  Specialist  they  work  all  the  D200  comps,  test  the  station 
and  work  all  the  TO  issue,  Item  managers  manage  different  parts  for  the  test  station  and 
make  sure  that  we  have  parts  in  supply,  Contracting  Officer  makes  that  the  contractor  is 
performing  its  duty  IAW  the  contract,  the  Engineer  is  responsible  for  the  software, 
drawings  and  support  the  ES. 

Participation  by  all  members  is  critical  to  successful  fielding  and  Sustainment  of  ATE. 
The  MAJCOMs  provide  valuable  insight  into  the  day  to  day  functional  test  requirements. 
Every  effort  should  be  made  to  have  them  during  the  initial  and  all  follow-on  phases  of 
acquisition  and  sustainment. 


10.  Are  you  familiar  with  DoD  policy  regarding  the  acquisition  and  development  of 
common  test  equipment?  (AFPD  63-2) 


No  I  am  not.  I  do  know  that  we  have  a  requirement  to  use  existing  equipment  to  cut 
down  on  proliferation  of  unique  testers. 


1 1 .  How  does  this  DoD  policy  affect  this  equipment,  if  at  all? 


Not  understanding  this  policy  I  am  not  sure  how  to  answer  this  question. 
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12.  Please  add  any  additional  infonnation  you  feel  would  aid  in  answering  Investigative 
Question  #1. _ 
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Appendix  K:  Selected  SPO  Data  for  IQ  2 


Investigative  Question  #2:  How  much  funding  is  budgeted/funded  in  the  Program 
Objective  Memorandum  (POM)  and  Future  Year  Defense  Plan  (FYDP)  for  your  ATS 
program? 


Sources  of  Data: 

MAJCOM  and  SPO/depot  financial  managers 
MAJCOM  and  SPO/depot  subject  matter  experts 
Budget  reports 

Budget  Estimate  Submissions  (BES) 


Supporting  Questions: 

(Please  type  answers  directly  below  each  question  and  use  as  much  space  as  required) 


1 .  Who  is  responsible  for  funding  your  related  equipment? 


F-15  SPO:  modifications,  production,  TCTOs,  initial  spares 
3600  development  and  3010  production  money 
LEACC:  handles  funding  for  repairs  and  spares 


2.  Are  there  any  funded/unfunded  requirements  for  the  equipment?  Please  explain. 


TTIP  Wholly  funded  with  3600/3010  money  through  2007 
Planning  process  for  ’08  POM 
Depot  money: 


583 

MSD  Eng 

MSD  UUT 

540 

545 

Repairs 

Spares 

FY01 

$ 

600,000.00 

$ 

- 

$ 

$ 

$ 

$ 

3,120,000.00 

$ 

350,000.00 

FY02 

$ 

600,000.00 

$ 

- 

$ 

$ 

$ 

$ 

3,120,000.00 

$ 

350,000.00 

FY03 

$ 

600,000.00 

$ 

1,030,000.00 

$ 

$ 

$ 

$ 

3,120,000.00 

$ 

350,000.00 

FY04 

$ 

635,000.00 

$ 

1,030,000.00 

$ 

$ 

$ 

$ 

3,192,000.00 

$ 

723,000.00 

FY05 

$ 

635,000.00 

$ 

3,550,000.00 

$ 

$ 

$ 

$ 

3,320,000.00 

$ 

370,000.00 

These  requirements  were  funded  100%  and  executed  for  all  years  except  the  MSD 
engineering  dollars  for  FY05.  Those  funds  are  in  limbo  because  program  changes 
required  a  change  in  our  original  justification.  The  change  generated  a  lot  of 
questions  and  second-guessing  by  the  funding  command  which  we  are  currently 
addressing. 


3.  What  equipment  funding  requirements  were  included  in  the  last  2  POM  cycles? 
Please  include  the  associated  BES  input. _ 


TTIP  inputs  were  first  POM  money  received  since  1991 
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First  tried  in  2000,  not  even  close 

In  2002,  didn’t  miss  the  cut  by  much 

Got  it  in  ’04;  one  of  only  two  F15  inputs  to  make  the  cut 

TTIP  received  $41.6M  from  the  2004  POM;  $36M  for  production  and 

development  contract,  remaining  funds  dedicated  to  engineering  support. 

The  $36M  will  be  evenly  distributed  between  3010  production  funds  and  3600 

_ development  funds. _ 

4.  Do  you  get  the  funds  needed  to  adequately  support  your  equipment  program?  Please 
explain. _ 

Yes,  right  now 

Important  to  plan  well  in  advance 

Preparation  is  key;  used  a  case  study  with  predicted  impact  to  justify 


5.  Are  your  equipment  POM  inputs  funded?  If  not,  how  far  below  the  Air 
Staff/MAJCOM  funding  line  did  the  requirements  fall? _ 


Yes 


6.  Please  add  any  additional  infonnation  you  feel  would  aid  in  answering  Investigative 
Question  #2. _ 
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Appendix  L:  Selected  Depot  Data  for  IQ  2 


Investigative  Question  #2:  How  much  funding  is  budgeted/funded  in  the  Program 
Objective  Memorandum  (POM)  and  Future  Year  Defense  Plan  (FYDP)  for  your  ATS 
program? 


Sources  of  Data: 

MAJCOM  and  SPO/depot  financial  managers 
MAJCOM  and  SPO/depot  subject  matter  experts 
Budget  reports 

Budget  Estimate  Submissions  (BES) 


Supporting  Questions: 

(Please  type  answers  directly  below  each  question  and  use  as  much  space  as  required) 


1 .  Who  is  responsible  for  funding  your  related  equipment? 


540  -  AFMC 

MSD  Engineering  -  AFMC 
583  -  ACC/ANG/AFRC 


2.  Are  there  any  funded/unfunded  requirements  for  the  equipment?  Please  explain. 


For  MSD  Engineering  normally  funded  by  approx  30%  of  actual  requirement. 
For  540  normally  funded  by  approx  80%  of  actual  requirement. 

For  583  normally  funded  by  approx  15%  of  actual  requirement. 


FY01 

FY02 

FY03 

FY04 

FY05 


3.  What  equipment  funding  requirements  were  included  in  the  last  2  POM  cycles? 
Please  include  the  associated  BES  input. 


None. 


4.  Do  you  get  the  funds  needed  to  adequately  support  your  equipment  program?  Please 
explain. 


583 _ MSD  Eng _ MSD  UUT _ 540 _ 545 


$ 

$  6,100,000.00 

$  9,400,000.00 

$ 

$ 

$ 

$  6,100,000.00 

$ 

$  118,000.00 

$ 

$ 

$  6,600,000.00 

$ 

$  1,600,000.00 

$ 

$  933,000.00 

$  4,200,000.00 

$ 

$  1,400,000.00 

$463,000.00 

$  6,000,000.00 

$  14,000,000.00 

$  7,000,000.00 

$  2,000,000.00 

$300,000.00 
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Inadequate  funding  to  properly  support  ATS  and  EARTS.  Need  funds  to  completely 
replace  the  whole  system  with  state  of  the  art  technology  to  ensure  the  user’s  mission 
capability  will  not  be  degraded.  Until  this  happens  the  operational  capability  of  the  AIS 
is  in  jeopardy  of  being  compromised  and  eventual  grounding  of  aircraft  will  occur. 


5.  Are  your  equipment  POM  inputs  funded?  If  not,  how  far  below  the  Air 
Staff/MAJCOM  funding  line  did  the  requirements  fall? _ 


I  don’t  think  MSD  Engineering  has  a  POM.  I  think  it  is  just  available  annually.  Put 
down  MSD  Engineering  $5M/yr  is  usually  issued.  I  asked  for  about  $18M  each  year  and 
they  approve  about  $5M. 

I  am  not  sure  if  540  (Software  Funds)  has  a  POM.  I  normally  get  about  $1M  and  ask  for 
about  $1.5M. 


6.  Please  add  any  additional  infonnation  you  feel  would  aid  in  answering  Investigative 
Question  #2. _ 
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Appendix  M:  Selected  SPO  Data  for  IQ  3 


Investigative  Question  #3:  What  are  ATE  SPO,  depot,  and  MAJCOM  assessments  of 
ATE/ATS  sustainability  for  the  two  programs? 

Sources  of  Data: 

MAJCOM  Staff 

SPO/depot  program  managers 

SPO/depot  equipment  specialists 

Supporting  Questions: 

(Please  type  answers  directly  below  each  question  and  use  as  much  space  as  required) 

1.  What  specific  sustainability  issues  (e.g.  hardware,  components)  does  this  equipment 
have? 

-  ELGAR 

Freq  Synthesizer:  COTS  with  unique  mods,  limited  number,  contractor  unable  to 
support  past  ’08. 

2.  Is  there  an  action  plan  to  address  these  issues?  Please  explain. 

Yes,  TTIP  looking  for  suitable  sub/development  work 

Plan  developed  by  internal  IPT 

IPT  is  key  to  address  these  issues 

3.  Are  these  issues  solvable?  If  not,  why?  What  roadblocks,  if  any,  are  hindrances  to 
reaching  a  solution? 

Yes 

No  concerns  assuming  expected  approval  for  funding  in  ’08  POM 

Optimistic  about  support  from  ACC 

4.  Are  there  timelines  for  reaching  a  solution?  Please  explain. 

The  SPO  received  funding  to  procure  18  of  the  new  Elgar  P/Ss  to  use  in 
workaround  packages  to  support  the  Field  until  the  TTIP  upgrade.  These  WA  (- 
2181)  packages  became  available  in  Aug  04 

5.  When  did  these  issues  first  arise? 

6.  How  urgent  of  a  priority  are  these  issues? 

7.  Please  add  any  additional  infonnation  you  feel  would  aid  in  answering  Investigative 

127 


Question  #3. _ 

MANUFACTURER  NAME  MASKED  has  core  group  of  engineers  since  day  1  of 

TISS  program  -  continuity  very  valuable 

Perform  “maintenance  assistance  visits”  to  aid  field  performance 

8.  Please  e-mail  completed  questionnaire  to  ieremy.howe@afit.edu.  Thank  you  for  your 
assistance. 
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Appendix  N:  Selected  Depot  Data  for  IQ  3 


Investigative  Question  #3:  What  are  ATE  SPO,  depot,  and  MAJCOM  assessments  of 
ATE/ATS  sustainability  for  the  two  programs? 


Sources  of  Data: 

MAJCOM  Staff 

SPO/depot  program  managers 

SPO/depot  equipment  specialists 

Supporting  Questions: 

(Please  type  answers  directly  below  each  question  and  use  as  much  space  as  required) 


1.  What  specific  sustainability  issues  (e.g.  hardware,  components)  does  this 
equipment  have? _ 


Operational  capability  approaching  50%  level,  (i multiple  respondents  indicated) 
System  Health  is  Poor 

Modified  once  (1985)  since  initial  fielding  in  the  70s 
Over  1 8  years  has  transpired  without  any  improvements 

Critical  TRUs  impacting  system  now 

Digital  Multimeter  (DMM) 

RMS  VOLTMETER 

HF  COUNTER 

Oscilloscope 

Spectrum  Analyzer 

X/L  -  Band  Signal  Generator 

Impedance  Unit  #1  &  #  2 

TRUs  are  unique  &  unsupportable 

Limited  or  no  repair  sources  exist 

Component  level  parts  are  non-procurable  or  repairable 

Limited  or  no  spares  in  the  inventory 

50%  below  spares  authorizations 

Reference  F-15  ATS  Slideshow  included  -AUTHOR  NAME  MASKED. 
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2.  Is  there  an  action  plan  to  address  these  issues?  Please  explain. 


Annual  requests  for  funding  in  the  583,  MSD  Engineering,  and  540  budget  cycles. 
Partial  funding  is  band  aiding  this  system. 


3.  Are  these  issues  solvable?  If  not,  why?  What  roadblocks,  if  any,  are  hindrances  to 
reaching  a  solution? 


Yes  with  adequate  funding. 


4.  Are  there  timelines  for  reaching  a  solution?  Please  explain. 


Projected  grounding  of  F-15s  in  FY07. 


5.  When  did  these  issues  first  arise? 


Ongoing  for  last  10-15  years.  We  have  been  continually  requesting  funds  to  replace  the 
ATS  and  later  on  the  EARTS  since  they  are  not  being  replaced  by  ESTS.  (multiple 
respondents  indicated) 


6.  How  urgent  of  a  priority  are  these  issues? 


Critical. 

Scenario  I 

Grounding  will  occur  through  normal  Obsolescence  in  CY07 
Scenario  II 

If  any  critical  TRU  fails  the  following  will  occur 
Lateral  support  between  I-level  will  be  required 

A/C  start  grounding  within  7-9  months  (or  sooner)  once  Spares  are  exhausted  or 
when  unit  is  unable  to  compete  with  local/lateral  demand 

Reference  F-15  ATS  Slideshow  included  -AUTHOR  NAME  MASKED. 


7.  Please  add  any  additional  information  you  feel  would  aid  in  answering  Investigative 
Question  #3. 


8.  Please  e-mail  completed  questionnaire  to  jeremy.howe@afit.edu.  Thank  you  for  your 
assistance. 
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Appendix  O:  Selected  MAJCOM  Data  for  IQ  3 


Investigative  Question  #3:  What  are  ATE  SPO,  depot,  and  MAJCOM  assessments  of 
ATE/ATS  sustainability  for  the  two  programs? 

Sources  of  Data: 

MAJCOM  Staff 

SPO/depot  program  managers 

SPO/depot  equipment  specialists 

Supporting  Questions: 

(Please  type  answers  directly  below  each  question  and  use  as  much  space  as  required) 

1.  What  specific  sustainability  issues  (e.g.  hardware,  components)  does  this  equipment 
have? 

Primarily,  the  equipment  is  plagued  with  obsolescence  and  diminishing  manufacturing 
sources  or  DMS. 

2.  Is  there  an  action  plan  to  address  these  issues?  Please  explain. 

Yes.  The  SPO  and  Warner  Robins  address  obsolescence  via  sustaining  engineering 
methods  whereby  they  evaluate  each  Test  Replaceable  Unit  (TRU)  on  the  test  stations  as 
well  as  internal  components.  After  evaluation,  the  engineers  attempt  to  resolve  issues  via 
Form,  Fit,  and  Function  replacements  and  or  seek  new  vendors  to  supply  components. 

3.  Are  these  issues  solvable?  If  not,  why?  What  roadblocks,  if  any,  are  hindrances  to 
reaching  a  solution? 

These  issues  are  solvable.  However,  the  primary  roadblock  in  a  timely  resolution  is 
funding. 

4.  Are  there  timelines  for  reaching  a  solution?  Please  explain. 

Timelines  are  established,  but  they  are  based  on  several  factors:  degree  of  obsolescence; 
availability  of  repair  sources;  logistical  impacts. 

5.  When  did  these  issues  first  arise? 

Obsolescence  is  a  nonnal  occurrence  and  usually  impacts  the  weapon  system  (were 
replacements  are  necessary)  within  7-10  years. 

6.  How  urgent  of  a  priority  are  these  issues? 

Depending  upon  the  impact  to  weapon  system  availability,  the  priority  can  be  urgent  but 
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normally,  if  managed  correctly,  is  routine. 

Researcher  Follow-up  clarification  regarding  “ managed  correctly” :  From  a  sustainment 
perspective,  Program  Management  and  Engineer  should  predict,  forecast,  and  continually 
monitory  the  health  of  their  system/program  to  ensure  obsolescence  issues  can  be 
addressed  in  a  timely  manor.  Otherwise  implementation  of  fixes  becomes  more  costly 
and  is  usually  late  to  need  therefore  becoming  urgent  in  nature. 


7.  Please  add  any  additional  infonnation  you  feel  would  aid  in  answering  Investigative 
Question  #3. _ 
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Appendix  P:  Selected  MAJCOM  Data  for  IQ  4 


Investigative  Question  #4:  What  are  MAJCOM  assessments  of  field  units’  abilities  to 
support  assigned  equipment  with  resources  provided  by  SPO/depot? 

Sources  of  Data: 

MAJCOM  subject  matter  experts 

Memorandums  documenting  ATE/ATS  problems 

REMIS  data 

Supporting  Questions: 

(Please  type  answers  directly  below  each  question  and  use  as  much  space  as  required) 

1 .  Do  units  get  the  parts  they  need  to  support  their  assigned  equipment?  Please  explain. 

Yes 

2.  Do  units  get  the  replacement  parts  in  a  timely  fashion?  Please  explain. 

Yes,  unless  there  is  unforeseen  supply/transportation  issues. 

3.  How  often  do  technicians  have  to  rely  on  technical  procedures  (at  the  direction  of  the 
SPO)  not  provided  in  established  TO  procedures?  Who  proposed  the  solution  (field, 
MAJCOM,  or  SPO/depot)?  Please  explain. 

Very  seldom  does  the  backshop  require  TO  direction  from  the  SPO.  If  needed,  the  SPO, 
depot,  and  MAJCOM  determine  solution  based  on  engineering  analysis/direction. 

4.  Are  TOs  of  good  quality?  Please  explain. 

Yes,  but  that  depends  on  the  unit’s  ability  to  maintain  proper  upkeep. 

5.  Are  TCTOs  of  good  quality?  Please  explain. 

Normally,  yes.  But  there  needs  a  high  level  of  coordination  between  MAJCOM,  SPO, 
and  depot  to  ensure  accuracy. 

6.  Do  units  receive  adequate  SPO  troubleshooting  support  when  formal  technical  data  is 
exhausted?  Please  explain. 

Yes.  Telephonic  support  is  established  for  all  major  systems  (TISS,  ANT/EARTS,  and 
ESTS).  For  TISS,  the  technicians  can  call  MANUFACTURER  NAME  MASKED  vfr 
direct,  for  ANT  they  can  call  depot  and  for  ESTS  they  can  call  MANUFACTURER 

NAME  MASKED. 
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7.  Are  there  relatively  more,  fewer,  or  about  the  same  number  of  MICAPs  as  in  the  past? 
Please  explain. _ 


They  are  probably  more  MICAPs  due  to  increased  Ops  TEMPO  and  due  to  aging  weapon 
systems. 


8.  Are  backorders  longer  than  30  days  common?  Give  examples  of  recurring  problems. 


If  you’re  referring  to  LRUs,  usually  we  can  get  replacements  well  under  30  days. 
However,  for  station  parts  its  120  days  for  older  weapon  systems. 


9.  Please  add  any  additional  infonnation  you  feel  would  aid  in  answering  Investigative 
Question  #4. _ 
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